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Why  GAO  Did  This  Study 

GAO  has  designated  the 
Department  of  Defense’s  (DOD) 
business  systems  modernization  as 
a  high-risk  program  because, 
among  other  things,  it  has  been 
challenged  in  implementing  key 
information  technology  (IT) 
management  controls  on  its 
thousands  of  business  systems.  The 
Global  Combat  Support  System- 
Marine  Corps  program  is  one  such 
system.  Initiated  in  2003,  the 
program  is  to  modernize  the 
Marine  Corps  logistics  systems. 

The  first  increment  is  to  cost  about 
$442  million  and  be  deployed  in 
fiscal  year  2010.  GAO  was  asked  to 
determine  whether  the  Department 
of  the  Navy  is  effectively 
implementing  IT  management 
controls  on  this  program.  To 
accomplish  this,  GAO  analyzed  the 
program’s  implementation  of 
several  key  IT  management 
disciplines,  including  economic 
justification,  earned  value 
management,  risk  management, 
and  system  quality  measurement. 


What  GAO  Recommends 


GAO  is  making  recommendations 
to  the  Secretary  of  Defense  aimed 
at  limiting  investment  in  the 
program  and  addressing  its  cost 
and  schedule  estimating,  risk 
management,  and  system  quality 
measurement  weaknesses.  DOD 
agreed  in  full  or  in  part  with  GAO’s 
recommendations  and  described 
ongoing  and  planned  actions 
intended  to  address  the 
recommendations. 


To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-822. 

For  more  information,  contact  Randolph  C. 
Hite  at  (202)  512-3439  or  hiter@gao.gov. 


What  GAO  Found 

DOD  has  not  effectively  implemented  key  IT  management  controls  provided 
for  in  DOD  and  related  acquisition  guidance  on  this  program.  If  implemented 
effectively,  these  and  other  IT  management  disciplines  increase  the  likelihood 
that  a  given  system  investment  will  produce  the  right  solution  to  fill  a  mission 
need  and  that  this  system  solution  will  be  acquired  and  deployed  in  a  manner 
that  maximizes  the  chances  of  delivering  promised  system  capabilities  and 
benefits  on  time  and  within  budget.  Neither  of  these  outcomes  is  being  fully 
realized  on  this  program,  as  evidenced  by  the  fact  that  its  first  increment  has 
already  slipped  more  than  3  years  and  is  expected  to  cost  about  $193  million 
more  than  envisioned.  These  slippages  and  cost  overruns  can  be  attributed  in 
part  to  the  management  control  weaknesses  discussed  in  this  report  and 
summarized  below.  Moreover,  additional  slippages  and  overruns  are  likely  if 
these  and  other  IT  management  weaknesses  are  not  addressed. 

•  Investment  in  the  system  has  not  been  economically  justified  on  the  basis 
of  reliable  estimates  of  both  benefits  and  costs.  Specifically,  while 
projected  benefits  were  risk-adjusted  to  compensate  for  limited  data  and 
questionable  assumptions,  the  cost  side  of  the  benefit/cost  equation  is  not 
sufficiently  reliable  because  it  was  not  derived  in  accordance  with  key 
cost  estimating  practices.  In  particular,  it  was  not  based  on  historical  data 
from  similar  programs  and  it  did  not  account  for  schedule  risks,  both  of 
which  are  needed  for  the  estimate  to  be  considered  accurate  and  credible. 

•  Earned  value  management  that  the  program  uses  to  measure  progress  has 
not  been  adequately  implemented.  Specifically,  the  schedule  baseline 
against  which  the  program  gauges  progress  is  not  based  on  key  estimating 
practices  provided  for  in  federal  guidance,  such  as  assessing  schedule 
risks  and  allocating  schedule  reserves  to  address  these  risks.  As  a  result, 
program  progress  cannot  be  adequately  measured,  and  likely  program 
completion  dates  cannot  be  projected  based  on  actual  work  performed. 

•  Some  significant  program  risks  have  not  been  adequately  managed.  While 
a  well-defined  risk  management  plan  and  supporting  process  have  been 
put  in  place,  the  process  has  not  always  been  followed.  Specifically, 
mitigation  steps  for  significant  risks  either  have  not  been  implemented  or 
proved  ineffective,  allowing  the  risks  to  become  actual  problems. 

•  The  data  needed  to  produce  key  indicators  of  system  quality,  such  as 
trends  in  the  volume  of  significant  and  unresolved  problems  and  change 
requests,  are  not  being  collected.  Without  such  data,  it  is  unclear  whether 
the  system  is  becoming  more  or  less  mature  and  stable. 

The  reasons  for  these  weaknesses  range  from  limitations  of  DOD  guidance 
and  tools,  to  not  collecting  relevant  data.  Until  they  are  addressed,  DOD  is  at 
risk  of  delivering  a  solution  that  does  not  cost-effectively  support  mission 
operations  and  falls  short  of  cost,  schedule,  and  capability  expectations. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


July  28,  2008 

The  Honorable  Daniel  K.  Akaka 
Chairman 

The  Honorable  John  Thune 
Ranking  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  John  Ensign 
United  States  Senate 

For  decades,  the  Department  of  Defense  (DOD)  has  been  challenged  in 
modernizing  its  timeworn  business  systems.1  In  1995,  we  designated  DOD’s 
business  systems  modernization  program  as  high  risk  and  continue  to  do 
so  today.2  Our  reasons  include  the  modernization’s  large  size,  complexity, 
and  critical  role  in  addressing  other  long-standing  transformation  and 
financial  management  challenges.  Other  reasons  are  that  DOD  has  yet  to 
institutionalize  key  system  modernization  management  controls,  and  it  has 
not  demonstrated  the  ability  to  consistently  deliver  promised  system 
capabilities  and  benefits  on  time  and  within  budget. 

Nevertheless,  DOD  continues  to  invest  billions  of  dollars  in  thousands  of 
business  systems,  including  about  a  hundred  that  the  department  has 
labeled  as  business  transformational  programs,  12  of  which  account  for 
about  50  percent  of  these  programs’  costs.  The  Global  Combat  Support 
System-Marine  Corps  (GCSS-MC)  is  one  such  program.  Initiated  in  2003, 
GCSS-MC  is  to  modernize  the  Marine  Corps  logistics  systems  and  thereby 
provide  decision  makers  with  timely  and  complete  logistics  information  to 
support  the  warfighter.  As  envisioned,  the  program  consists  of  a  series  of 
major  increments,  the  first  of  which  is  expected  to  cost  approximately 
$442  million  and  be  fully  deployed  in  fiscal  year  2010. 


business  systems  are  information  systems,  including  financial  and  nonfinancial  systems, 
that  support  DOD  business  operations,  such  as  civilian  personnel,  finance,  health,  logistics, 
military  personnel,  procurement,  and  transportation. 

2GAO,  High-Risk  Series:  An  Update,  GAO-07-310  (Washington,  D.C.:  January  2007). 
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As  agreed,  our  objective  was  to  determine  whether  the  Department  of  the 
Navy  is  effectively  implementing  information  technology  (IT)  management 
controls  on  GCSS-MC.  To  accomplish  this,  we  focused  on  the  first 
increment  of  GCSS-MC  by  analyzing  a  range  of  program  documentation 
and  interviewing  cognizant  officials  relative  to  the  following  management 
areas:  architectural  alignment,  economic  justification,  earned  value 
management,  requirements  management,  risk  management,  and  system 
quality  measurement.  We  conducted  our  performance  audit  from  June 
2007  to  July  2008,  in  accordance  with  generally  accepted  government 
auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objective.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objective.  Additional  details 
on  our  objective,  scope,  and  methodology  are  in  appendix  I. 


Results  in  Brief 


DOD  has  not  effectively  implemented  key  IT  management  controls  on  its 
GCSS-MC  program.  Collectively,  these  management  controls  are  intended 
to  reasonably  ensure  that  investment  in  a  given  system  represents  the  right 
solution  to  fill  a  mission  need — and  if  it  is,  that  the  system  is  acquired  and 
deployed  the  right  way,  meaning  that  it  is  done  in  a  manner  that  maximizes 
the  chances  of  delivering  defined  system  capabilities  and  benefits  on  time 
and  within  budget.  Given  that  deployment  of  GCSS-MC  is  more  than  3 
years  behind  schedule  and  expected  to  cost  about  $193  million  more  than 
envisioned,  these  goals  are  already  not  being  met,  in  part  because  DOD 
program  management  and  oversight  entities  have  not  adequately 
implemented  several  key  IT  management  controls.  As  a  result,  the 
department  does  not  have  a  sufficient  basis  for  knowing  that  GCSS-MC,  as 
defined,  is  the  best  system  solution  to  meeting  its  mission  needs,  and  the 
program  is  likely  to  experience  further  schedule  slips  and  cost  overruns, 
along  with  reduced  system  capabilities.  Weaknesses  associated  with 
DOD’s  implementation  of  five  key  IT  management  controls,  as  well  as 
recent  actions  to  correct  weaknesses  with  another  management  control, 
are  as  follows: 

GCSS-MC  compliance  with  DOD’s  federated  business  enterprise 
architecture  (BEA)  has  not  been  sufficiently  demonstrated.  To  its  credit, 
the  program  office  has  followed  DOD’s  BEA  compliance  guidance. 
However,  the  program’s  compliance  assessment  (1)  did  not  include  all 
relevant  architecture  products,  such  as  those  that  describe  technical  and 
system  elements;  (2)  was  not  used  to  identify  potential  areas  of 
duplication  across  programs;  and  (3)  did  not  address  compliance  with  the 
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Department  of  the  Navy’s  enterprise  architecture.  These  important  steps 
were  not  performed  because  of  policy,  guidance,  and  tool  limitations,  and 
because  aspects  of  the  corporate  BEA  and  the  Department  of  the  Navy’s 
enterprise  architecture,  which  are  both  major  components  of  DOD’s 
federated  BEA,  have  yet  to  be  sufficiently  defined  to  permit  thorough 
compliance  determinations  in  these  areas.  In  addition,  program  oversight 
and  approval  authorities  did  not  validate  the  program  office’s  compliance 
assessments.  As  a  result,  the  department  does  not  have  a  sufficient  basis 
for  knowing  if  GCSS-MC  has  been  defined  to  optimize  the  DOD  and 
Department  of  the  Navy  business  operations. 

•  Investment  in  GCSS-MC  has  not  been  economically  justified.  According  to 
the  program’s  economic  analysis,  the  first  increment  will  have  an 
estimated  life  cycle  cost  of  about  $442  million  and  deliver  about  $1.04 
billion  in  risk-adjusted,  estimated  benefits.  While  the  most  recent  cost 
estimate  was  derived  using  some  effective  estimating  practices,  it  was  not 
based  on  other  practices  that  are  essential  to  having  an  accurate  and 
credible  estimate.  For  example,  the  estimate  was  not  grounded  in  a 
historical  record  of  comparable  data  from  similar  programs  and  did  not 
account  for  significant  risks  associated  with  the  program’s  aggressive 
schedule,  both  of  which  limit  the  estimate’s  accuracy  and  credibility. 

These  important  practices  were  not  employed  for  various  reasons, 
including  a  lack  of  historical  data  from  similar  programs  and  a  schedule 
risk  analysis  to  assess  the  cost  estimate’s  variability.  As  a  result,  an 
adequate  basis  for  informed  investment  decision  making  does  not  exist, 
and  actual  program  costs  will  likely  not  be  consistent  with  estimates. 

•  Earned  value  management  (EVM),  which  is  a  means  for  determining  and 
disclosing  actual  program  cost  and  schedule  performance  in  comparison 
with  estimates,  is  not  being  effectively  performed  because  the  schedule 
baseline  is  not  reliable.  Specifically,  while  the  program’s  current  schedule 
baseline  was  derived  using  some  key  estimating  practices,  it  is  not 
reflective  of  other  important  practices,  such  as  conducting  a  schedule  risk 
assessment  and  allocating  schedule  reserve.  These  important  practices 
were  not  followed,  according  to  program  officials,  because  doing  so  would 
have  pushed  back  the  estimated  completion  date  for  the  first  increment, 
which  they  said  would  not  have  been  consistent  with  DOD  oversight  and 
approval  authorities’  direction  to  complete  the  increment  as  soon  as 
possible.  In  other  words,  the  program  office  adopted  what  amounted  to  an 
imposing  completion  date  but  did  not  adjust  the  scope  and  schedule  of 
work  to  be  completed  to  make  this  date  attainable.  The  result  is  a  schedule 
that  is  not  reliable  and  does  not  provide  a  sufficient  baseline  for 
performing  EVM. 
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•  Despite  limitations  in  earlier  efforts  to  manage  GCSS-MC’s  requirements, 
improvements  have  since  been  made.  During  the  initial  phase  of  our 
review,  the  program  office  could  not  trace  all  of  its  1,375  system-level 
requirements  to  design  specifications  and  test  documentation.  For 
example,  about  30  percent  of  the  program’s  design  documents  had  yet  to 
be  validated,  approved,  and  linked  to  the  requirements.  This  was 
significant  because  it  had  already  contributed  to  lengthy  delays  to  the 
program’s  schedule  and,  without  adequate  traceability,  the  risk  of  the 
system  not  performing  as  intended  and  requiring  expensive  rework  is 
increased.  Following  our  inquiries  into  the  traceability  process  being  used, 
program  officials  changed  the  process.  Since  then,  our  analysis  of  61 
randomly  selected,  system-level  requirements  confirmed  that  60  are  now 
traceable  backward  to  operational  requirements  and  forward  to  design 
specifications  and  test  plans.  If  implemented  effectively,  the  new  process 
should  address  previous  requirements’  traceability  weaknesses  and 
thereby  avoid  a  repeat  of  past  problems. 

•  Despite  having  a  well-defined  process  for  managing  program  risks,  not  all 
program  risks  have  been  adequately  managed.  For  example,  of  25  medium 
risks  that  the  program  office  reported  as  closed  as  of  February  2008,  4 
were  closed  because  they  became  actual  issues  (problems).  In  each  of 
these  cases,  the  risk  mitigation  strategy  was  not  fully  implemented.  In 
addition,  4  were  closed  because,  even  though  the  risk  strategies  were 
implemented,  the  strategies  did  not  mitigate  the  risks,  resulting  in  each 
becoming  an  actual  problem.  Program  officials  attributed  the  lack  of 
success  in  mitigating  these  risks  to,  among  other  things,  resource 
constraints  and  an  aggressive  program  schedule.  Unless  program  risks  are 
effectively  mitigated,  GCSS-MC  will  experience  further  cost,  schedule,  and 
performance  shortfalls. 

•  Sufficient  data  for  measuring  trends  in  the  number  of  high  priority, 
unresolved  system  issues  (problems)  and  system  change  requests,  both  of 
which  are  recognized  indicators  of  a  system’s  quality  or  maturity,  are  not 
available.  On  the  positive  side,  the  program  office  has  established 
processes  for  (1)  collecting  and  tracking  data  on  the  status  of  program 
issues,  including  problems  discovered  during  early  test  events,  and  (2) 
capturing  data  on  the  status  of  requests  for  changes  to  the  system. 
Moreover,  program  documentation  emphasizes  the  importance  of 
monitoring  trends  in  program  problems  and  change  requests  that  have  yet 
to  be  addressed  or  resolved.  However,  an  effective  means  for  producing 
the  full  complement  of  data  that  are  needed  for  a  reliable  picture  of  such 
trends  does  not  exist.  In  particular,  data  on  problems  and  change  request 
priority  levels  and  closure  dates  are  not  consistently  maintained,  and 
program  oversight  of  contractor-identified  issues  or  defects  is  limited. 
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Program  officials  stated  that  they  intend  to  address  these  data  limitations, 
but  stated  that  oversight  of  contractor-identified  issues  is  not  their 
responsibility.  Without  sufficient  data  available  to  understand  trends  in  the 
volume  and  severity  of  program  problems  and  changes,  it  is  unclear 
whether  GCSS-MC’s  quality  and  stability  are  moving  in  the  right  direction. 

Until  the  above  discussed  IT  management  controls  are  effectively 
implemented,  DOD  is  at  risk  of  investing  in  incremental  GCSS-MC  system 
solutions  that  do  not  optimally  support  corporate  mission  needs  and 
mission  performance,  and  do  not  satisfy  defined  requirements  and  meet 
schedule  and  cost  commitments.  These  risks  have  already  been  realized, 
as  evidenced  by  the  more  than  3-year  delay  and  $193  million  increase  in 
the  expected  costs  to  deploy  the  first  system  increment.  Accordingly,  we 
are  making  recommendations  to  the  Secretary  of  Defense  aimed  at 
ensuring  that  any  decision  to  invest  in  the  next  acquisition  phase  of  GCSS- 
MC’s  first  increment  is  made  in  light  of  the  status  of  efforts  to  address  the 
risks  discussed  in  this  report  and  that  investment  in  all  future  increments 
be  conditional  upon  the  program  having  fully  addressed  the  control 
weaknesses  discussed  in  this  report.  We  are  also  making 
recommendations  intended  to  correct  the  cost  estimating,  schedule 
estimating,  risk  management,  and  system  quality  measurement  control 
weaknesses  discussed  in  this  report.  However,  we  are  not  making 
recommendations  in  this  report  relative  to  addressing  the  architecture 
compliance  weaknesses  because  we  have  work  under  way  that  is  more 
broadly  focused  on  this  area  across  multiple  programs,  and  we  will  be 
making  recommendations  in  that  report.  We  are  also  not  making 
recommendations  regarding  requirements  traceability  because  the 
weaknesses  that  we  found  have  recently  been  corrected. 

In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Business  Transformation)  and  reprinted  in  appendix 
II,  the  department  stated  that  it  concurred  with  two  of  our 
recommendations  and  partially  concurred  with  the  remaining  five.  In 
general,  it  partially  concurred  with  the  five  recommendations  because  it 
said  that  efforts  were  either  under  way  or  planned  that  will  address  some 
of  the  weaknesses  that  these  recommendations  are  aimed  at  correcting. 

We  support  these  efforts  because  they  are  generally  consistent  with  the 
intent  of  the  recommendations  and  believe  that,  if  fully  and  properly 
implemented,  they  will  go  a  long  way  in  addressing  the  management 
control  weaknesses  that  our  recommendations  are  intended  to  correct. 
DOD’s  comments  are  reprinted  in  their  entirety  in  appendix  II  of  this 
report. 
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Background 


The  Department  of  the  Navy  (DON)  is  a  major  component  of  DOD, 
consisting  of  two  uniformed  services:  the  Navy  and  the  Marine  Corps.  The 
Marine  Corps’  primary  mission  is  to  serve  as  a  “total  force  in  readiness”  by 
responding  quickly  in  a  wide  spectrum  of  responsibilities,  such  as  attacks 
from  sea  to  land  in  support  of  naval  operations,  air  combat,  and  security  of 
naval  bases.  As  the  only  service  that  operates  in  three  dimensions — in  the 
air,  on  land,  and  at  sea,  the  Marine  Corps  must  be  equipped  to  provide 
rapid  and  precise  logistics3  support  to  operating  forces  in  any 
environment. 

The  Marine  Corps’  many  and  dispersed  organization  components  rely 
heavily  on  IT  to  perform  their  respective  mission-critical  operations  and 
related  business  functions,  such  as  logistics  and  financial  management. 

For  fiscal  year  2008,  the  Marine  Corps  budget  for  IT  business  systems  is 
about  $1.3  billion,  of  which  $746  million  (57  percent)  is  for  operations  and 
maintenance  of  existing  systems  and  $553  million  (43  percent)  is  for 
systems  development  and  modernization.  Of  the  approximately  904 
systems  in  DON’s  current  inventory,  the  Marine  Corps  accounts  for  81,  or 
about  9  percent,  of  the  total.  The  GCSS-MC  is  one  such  system  investment. 
According  to  DOD,  it  is  intended  to  address  the  Marine  Corps’  long¬ 
standing  problem  of  stove-piped  logistics  systems  that  collectively  provide 
limited  data  visibility  and  access,  are  unable  to  present  a  common, 
integrated  logistics  picture  in  support  of  the  warfighter,  and  do  not 
provide  important  decision  support  tools. 


GCSS-MC:  A  Brief 
Description 


In  September  2003,  the  Marine  Corps  initiated  GCSS-MC4  to  (1)  deliver 
integrated  functionality  across  the  logistics  areas  (e.g.,  supply  and 
maintenance),  (2)  provide  timely  and  complete  logistics  information  to 


3DOD  defines  logistics  as  the  science  of  planning  and  carrying  out  the  movement  and 
maintenance  of  forces.  Logistics  includes  the  aspects  of  military  operations  that  deal  with: 
(1)  design  and  development,  acquisition,  storage,  movement,  distribution,  maintenance, 
evacuation,  and  disposition  of  materiel;  (2)  movement,  evacuation,  and  hospitalization  of 
personnel;  (3)  acquisition  or  construction,  maintenance,  operation,  and  disposition  of 
facilities;  and  (4)  acquisition  or  furnishing  of  services. 

4GCSS-MC  was  formally  designated  a  Major  Automated  Information  System  Acquisition 
program  in  July  2004,  which  is  a  program  or  initiative  that  is  so  designated  by  the  Assistant 
Secretary  of  Defense  (Networks  and  Information  Integration)/Chief  Information  Officer  or 
that  is  estimated  to  require  program  costs  in  any  single  year  in  excess  of  $32  million  (fiscal 
year  2000  constant  dollars),  total  program  costs  in  excess  of  $126  million  (fiscal  year  2000 
constant  dollars),  or  total  life  cycle  costs  in  excess  of  $378  million  (fiscal  year  2000 
constant  dollars). 
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authorized  users  for  decision  making,  and  (3)  provide  access  to  logistics 
information  and  applications  regardless  of  location.  The  system  is 
intended  to  function  in  three  operational  environments — deployed 
operations  (i.e.,  in  theater  of  war  or  exercise  environment  on  land  or  at 
sea),  in-transit,  and  in  garrison.5  When  GCSS-MC  is  fully  implemented,  it  is 
to  support  about  33,000  users  located  around  the  world. 

GCSS-MC  is  being  developed  in  a  series  of  large  and  complex  increments 
using  commercially  available  enterprise  resource  planning  (ERP)6 
software  and  hardware  components.  The  first  increment  is  currently  the 
only  funded  portion  of  the  program  and  is  to  provide  a  range  of  asset 
management  capabilities,  including 

•  planning  inventory  requirements  to  support  current  and  future 
demands; 

•  requesting  and  tracking  the  status  of  products  (e.g.,  supplies  and 
personnel)  and  services  (e.g.,  maintenance  and  engineering); 

•  allocating  resources  (e.g.,  inventory,  warehouse  capacity,  and 
personnel)  to  support  unit  demands  for  specific  products;  and 

•  scheduling  maintenance  resources  (e.g.,  manpower,  equipment,  and 
supplies)  for  specific  assets,  such  as  vehicles. 

Additionally,  the  first  increment  is  to  replace  four  legacy  systems 
scheduled  for  retirement  in  2010.  Table  1  describes  these  four  systems. 


5 A  garrison  location  is  a  home  station,  usually  in  the  continental  United  States,  for  a  unit 
that  is  not  deployed. 

6An  ERP  solution  is  commercial  off-the-shelf  software  package  consisting  of  multiple, 
integrated  functional  modules  that  perform  a  variety  of  business  related  tasks,  such  as 
payroll,  general  ledger  accounting,  and  supply  chain  management. 
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Table  1:  Description  of  Legacy  Systems  Scheduled  for  Retirement  in  2010 

System 

Description 

Supported  Activities  and  Supply  System 

Thirty-five  year  old  mainframe  ground  supply  system  that  is  used  for  procuring, 
distributing,  and  managing  Marine  Corps’  supplies. 

Marine  Corps  Integrated  Maintenance 
Management  System 

Twenty-nine  year  old  mainframe  system  that  is  used  for  maintaining  ground 
equipment,  including  planning  and  managing  work  orders  and  parts,  and  for 
performing  equipment  readiness  reporting  and  status  tracking. 

Asset  Tracking  and  Logistics  System 

Fifteen-year-old  data  entry  system  dedicated  to  supporting  the  Supported  Activities 
and  Supply  System  and  the  Marine  Corps  Integrated  Maintenance  Management 
System  for  controlling,  distributing,  and  replenishing  equipment  and  supplies  in 
assigned  areas  of  operation  and  receiving  supply  support  from  and  providing  it  to 
other  military  departments. 

Personal  Computer  Marine  Corps  Integrated 
Maintenance  Management  System 

Fourteen-year-old  stand-alone  system  that  is  used  for  performing  a  limited  number  of 
ground  maintenance  management  logistics  functions. 

Source:  DOD. 

Future  increments  are  to  provide  additional  functionality  (e.g., 
transportation  and  wholesale  inventory  management),  enhance  existing 
functionality,  and  potentially  replace  up  to  44  additional  legacy  systems. 

The  program  office  estimates  the  total  life  cycle  cost  for  the  first 
increment  to  be  about  $442  million,  including  $169  million  for  acquisition 
and  $273  million  for  operations  and  maintenance.7  The  total  life  cycle  cost 
of  the  entire  program  has  not  yet  been  determined  because  future 
increments  are  currently  in  the  planning  stages  and  have  not  been  defined. 
As  of  April  2008,  the  program  office  reported  that  approximately  $125 
million  has  been  spent  on  the  first  increment. 


Program  Oversight  and 
Management  Roles  and 
Responsibilities 


To  manage  the  acquisition  and  deployment  of  GCSS-MC,  the  Marine  Corps 
established  a  program  management  office  within  the  Program  Executive 
Office  for  Executive  Information  Systems.  The  program  office  is  led  by  the 
Program  Manager  who  is  responsible  for  managing  the  program’s  scope 
and  funding  and  ensuring  that  the  program  meets  its  objectives.  To 


The  current  life  cycle  cost  estimate  is  from  fiscal  year  2007  through  fiscal  year  2019,  in 
base  year  2007  dollars,  and  excludes  costs  associated  with  supporting  and  maintaining 
legacy  systems  during  GCSS-MC  development,  totaling  $8.2  million,  and  does  not  reflect 
program  costs  of  $61.4  million  from  fiscal  year  2004  through  fiscal  year  2006  that  are 
considered  sunk  costs.  According  to  the  Office  of  Management  and  Budget,  sunk  costs  are 
those  incurred  in  the  past  that  will  not  be  affected  by  any  present  or  future  decision  and 
should  be  ignored  when  determining  whether  a  new  investment  is  worthwhile. 
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accomplish  this,  the  program  office  is  responsible  for  key  acquisition 
management  controls,  such  as  architectural  alignment,  economic 
justification,  EVM,  requirements  management,  risk  management,  and 
system  quality  measurement.  In  addition,  various  DOD  and  DON 
organizations  share  program  oversight  and  review  activities  relative  to 
these  and  other  acquisition  management  controls.  A  listing  of  key  entities 
and  their  roles  and  responsibilities  is  in  table  2. 

Table  2:  Organizations  Responsible  for  GCSS-MC  Oversight  and  Management 

Entity 

Roles  and  responsibilities 

Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Loqistics 
(USD  AT&L) 

Serves  as  the  milestone  decision  authority  (MDA),  which  according  to  DOD,  has  overall 
responsibility  for  the  program,  to  include  approving  the  program  to  proceed  through  its 
acquisition  cycle  on  the  basis  of,  for  example,  the  acquisition  plan,  an  independently 
evaluated  economic  analysis,  and  the  Acquisition  Program  Baseline. 

Assistant  Secretary  of  the  Navy, 

Research,  Development,  and  Acquisition 

Serves  as  DON’S  oversight  organization  for  the  program,  to  include  enforcement  of  USD 
AT&L  policies  and  procedures. 

Department  of  the  Navy,  Program 
Executive  Office  for  Executive 

Information  Systems 

Oversees  a  portfolio  of  large-scale  projects  and  programs  designed  to  enable  common 
business  processes  and  provide  standard  capabilities,  to  include  reviewing  the  acquisition 
plan,  economic  analysis,  and  the  Acquisition  Program  Baseline  prior  to  approval  by  the 

MDA. 

Department  of  the  Navy  Chief 

Information  Officer  (CIO) 

Supports  the  department’s  planning,  programming,  budgeting,  and  execution  processes  by 
ensuring  that  the  program  has  achievable  and  executable  goals  and  conforms  to  financial 
management  regulations,  and  DON,  DOD,  and  federal  IT  policies  in  several  areas  (e.g., 
security,  architecture,  and  investment  management);  it  also  works  closely  with  the  program 
office  during  milestone  review  assessments. 

Office  of  the  Secretary  of  Defense,  Office 
of  the  Director  for  Program  Analysis  and 
Evaluation 

Verifies  and  validates  the  reliability  of  cost  and  benefit  estimates  in  economic  analyses  and 
provides  its  results  to  the  MDA. 

Naval  Center  for  Cost  Analysis 

Performs  independent  program  cost  estimates. 

Defense  Cost  and  Resource  Center 

Collects  current  and  historical  major  defense  acquisition  program  cost  and  software 
resource  data  in  a  joint  service  environment  and  makes  those  data  available  for  use  by 
authorized  government  analysts  when  estimating  the  cost  of  ongoing  and  future 
government  programs. 

Deputy  Commandant,  Installations  and 
Logistics,  Headquarters,  Marine  Corps 

Provides  guidance  and  direction  for  overall  logistics  modernization  effort  and 
develops/approves  formal  capability  requirements  for  the  program. 

Defense  Business  Systems  Management 
Committee  (DBSMC) 

Serves  as  the  highest  ranking  governance  body  for  business  systems  modernization 
activities  and  approves  investments  costing  more  than  $1  million,  as,  for  example,  being 
compliant  with  the  BEA. 

Investment  Review  Board  (1 RB) 

Reviews  business  system  investments  and  has  responsibility  for  recommending 
certification  for  all  business  system  investments  costing  more  than  $1  million  that  are 
asserted  as  compliant  with  the  BEA. 

Business  Transformation  Agency  (BTA) 

Coordinates  business  transformation  efforts  across  DOD  and  supports  the  IRBs  and 

DBSMC. 
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Entity 

Roles  and  responsibilities 

GCSS-MC  Program  Management  Office 

Performs  day-to-day  program  management  and,  as  such,  is  the  single  point  of 
accountability  for  managing  the  program’s  objectives  through  development,  production, 
and  sustainment,  such  as  risk  management  and  coordinating  all  testing  activities  with 
requirements,  including  measuring  system  quality  and  managing  change  requests. 

Source:  DOD. 

Overview  of  GCSS-MC’s  The  program  reports  that  the  first  increment  of  GCSS-MC  is  currently  in 
Status  the  system  development  and  demonstration  phase  of  the  defense 

acquisition  system  (DAS).8  The  DAS  consists  of  five  key  program  life  cycle 
phases  and  three  related  milestone  decision  points.  These  five  phases  and 
related  milestones  are  described  along  with  a  summary  of  key  program 
activities  completed  during,  or  planned,  for  each  phase  as  follows: 

1.  Concept  refinement:  The  purpose  of  this  phase  is  to  refine  the  initial 
system  solution  (concept)  and  create  a  strategy  for  acquiring  the 
investment  solution.  During  this  phase,  the  program  office  defined  the 
acquisition  strategy  and  analyzed  alternative  solutions.  The  first 
increment  completed  this  phase  on  July  23,  2004,  which  was  1  month 
later  than  planned,  and  the  MDA  approved  a  Milestone  A  decision  to 
move  to  the  next  phase. 

2.  Technology  development:  The  purpose  of  this  phase  is  to  determine 
the  appropriate  set  of  technologies  to  be  integrated  into  the  investment 
solution  by  iteratively  assessing  the  viability  of  various  technologies 
while  simultaneously  refining  user  requirements.  During  this  phase, 
the  program  office  selected  Oracle’s  E-Business  Suite9  as  the 
commercial  off-the-shelf  ERP  software.  In  addition,  the  program  office 
awarded  Accenture  the  system  integration  contract  to,  among  other 
things,  configure  the  software,  establish  system  interfaces,  and 
implement  the  new  system.  This  system  integration  contract  was 
divided  into  two  phases — Part  1  for  the  planning,  analysis,  and 
conceptual  design  of  the  solution  and  Part  2  for  detailed  design,  build, 
test,  and  deployment  of  the  solution.  The  program  office  did  not 


8DAS  is  a  framework-based  approach  that  is  intended  to  translate  mission  needs  and 
requirements  into  stable,  affordable,  and  well-managed  acquisition  programs. 

9E-business  Suite  is  a  Web-based  software  system  consisting  of  various  software  (packaged 
software,  applications  server,  Web  server,  database  server  and  administrative  software) 
and  hardware  (servers,  switches,  storage,  and  ancillary  equipment)  to  support  the 
computing  environment. 
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exercise  the  option  for  Part  2  of  the  contract  to  Accenture  and  shortly 
thereafter  established  a  new  program  baseline  in  June  2006.  In 
November  2006,  it  awarded  a  time-and-materials  system  integration 
contract10  valued  at  $28.4  million  for  solution  design  to  Oracle.  The  first 
increment  completed  this  phase  on  June  8,  2007,  which  was  26  months 
later  than  planned  due  in  part  to  contractual  performance  shortfalls, 
and  the  MDA  approved  a  Milestone  B  decision  to  move  to  the  next 
phase. 

3.  System  development  and  demonstration:  The  purpose  of  this  phase  is 
to  develop  the  system  and  demonstrate  through  developer  testing  that 
the  system  can  function  in  its  target  environment.  During  this  phase, 
the  program  office  extended  the  solution  design  contract  and 
increased  funding  to  $67.5  million  due,  in  part,  to  delays  in  completing 
the  detailed  design  activities.  As  a  result,  the  program  office  has  not 
yet  awarded  the  next  contract  (which  includes  both  firm-fixed-price 
and  time-and-materials  task  orders)  for  build  and  testing  activities, 
originally  planned  for  July  2007.  Instead,  it  entered  what  it  termed  a 
“transition  period”  to  complete  detailed  design  activities.  According  to 
the  program’s  baseline,  the  MDA  is  expected  to  approve  a  Milestone  C 
decision  to  move  to  the  next  phase  in  October  2008.  However,  program 
officials  stated  that  Milestone  C  is  now  scheduled  for  April  2009,  which 
is  35  months  later  than  originally  planned. 

4.  Production  and  deployment:  The  purpose  of  this  phase  is  to  achieve 
an  operational  capability  that  satisfies  the  mission  needs,  as  verified 
through  independent  operational  test  and  evaluation,  and  implement 
the  system  at  all  applicable  locations.  The  program  office  plans  to 
award  a  separate  firm-fixed-price  plus  award  fee  contract11  for  these 
activities  with  estimated  costs  yet  to  be  determined. 

5.  Operations  and  support:  The  purpose  of  this  phase  is  to  operationally 
sustain  the  system  in  the  most  cost-effective  manner  over  its  life  cycle. 
The  details  of  this  phase  have  not  yet  been  defined. 


10Time-and-materials  contracts  provide  for  acquiring  supplies  or  services  on  the  basis  of  (1) 
direct  labor  hours  at  specified  fixed  hourly  rates  that  include  wages,  overhead,  general  and 
administrative  expenses,  and  profit  and  (2)  actual  cost  for  materials. 

nFixed-price  contracts  with  award  fees  include  a  fixed  priced  (including  normal  profit)  and 
an  additional,  separate  award  fee  amount.  The  fixed  price  is  paid  for  satisfactory 
performance;  the  award  fee,  if  any,  is  earned  based  on  periodic  evaluation  by  the 
government. 
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Overall,  GCSS-MC  was  originally  planned  to  reach  full  operational 
capability  (FOC)12  in  fiscal  year  2007  at  an  estimated  cost  of  about  $126 
million  over  a  7-year  life  cycle.13  This  cost  estimate  was  later  revised  in 
2006  to  about  $249  million  over  a  13-year  life  cycle.14  However,  the 
program  now  expects  to  reach  FOC  in  fiscal  year  2010  at  a  cost  of  about 
$442  million  over  a  12-year  life  cycle.  Figures  1  and  2  show  the  program’s 
current  status  against  original  milestones  and  original,  revised,  and  current 
cost  estimates. 


Figure  1:  GCSS-MC  Program  Schedule  Status 
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Source:  GAO  analysis  of  Marine  Corps  data. 


12FOC  means  that  the  system  has  been  deployed  in  all  intended  locations. 

13According  to  the  May  10,  2004,  analysis  of  alternatives,  this  estimate  was  a  “rough  order 
of  magnitude”  for  research  and  development,  procurement  and  operations  and  support 
from  fiscal  years  2004  through  2011. 

14According  to  the  July  15,  2005,  economic  analysis,  program  costs  are  estimated  from 
fiscal  years  2005  through  2018,  in  base  year  2005  dollars,  and  exclude  $9.6  million 
associated  with  supporting  and  maintaining  legacy  systems  during  GCSS-MC  development 
and  $11.9  million  in  fiscal  year  2004  sunk  costs. 
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Figure  2:  GCSS-MC  Program  Cost  Status 
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Source:  GAO  analysis  of  Marine  Corps  data. 


Use  of  IT  Acquisition 
Management  Best 
Practices  Maximizes 
Chances  for  Success 


Acquisition  best  practices  are  tried  and  proven  methods,  processes, 
techniques,  and  activities  that  organizations  define  and  use  to  minimize 
program  risks  and  maximize  the  chances  of  a  program’s  success.  Using 
best  practices  can  result  in  better  outcomes,  including  cost  savings, 
improved  service  and  product  quality,  and  a  better  return  on  investment. 
For  example,  two  software  engineering  analyses  of  nearly  200  systems 
acquisition  projects  indicate  that  teams  using  systems  acquisition  best 
practices  produced  cost  savings  of  at  least  11  percent  over  similar  projects 
conducted  by  teams  that  did  not  employ  the  kind  of  rigor  and  discipline 
embedded  in  these  practices.15  In  addition,  our  research  shows  that  best 


15Donald  E.  Harter,  Mayuram  S.  Krishnan,  and  Sandra  A.  Slaughter,  “Effects  of  Process 
Maturity  on  Quality,  Cycle  Time,  and  Effort  in  Software  Product  Development,” 
Management  Science,  46,  no.  4,  2000;  and  Bradford  K.  Clark,  “Quantifying  the  Effects  of 
Process  Improvement  on  Effort,”  IEEE  Software  (November/December  2000). 
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practices  are  a  significant  factor  in  successful  acquisition  outcomes  and 
increase  the  likelihood  that  programs  and  projects  will  be  executed  within 
cost  and  schedule  estimates.16 

We  and  others  have  identified  and  promoted  the  use  of  a  number  of  best 
practices  associated  with  acquiring  IT  systems.17  See  table  3  for  a 
description  of  several  of  these  activities. 

Table  3:  Summary  of  Selected  System  Acquisition  Best  Practices 

Business  practice 

Description 

Architectural  alignment 

To  ensure  that  the  acquisition  is  consistent  with  the 
organization’s  enterprise  architecture. 

Architectural  alignment  is  the  process  for  analyzing  and  verifying  that  the 
proposed  architecture  of  the  system  being  acquired  is  consistent  with  the 
enterprise  architecture  for  the  organization  acquiring  the  system.  Such  alignment 
is  needed  to  ensure  that  acquired  systems  can  interoperate  and  are  not 
unnecessarily  duplicative  of  one  another. 

Economic  justification 

To  ensure  that  system  investments  have  an 
adequate  economic  justification. 

Economic  justification  is  the  process  for  ensuring  that  acquisition  decisions  are 
based  on  reliable  analyses  of  the  proposed  investment’s  likely  costs  versus 
benefits  over  its  useful  life,  as  well  as  an  analysis  of  the  risks  associated  with 
actually  realizing  the  acquisition’s  forecasted  benefits  for  its  estimated  costs. 
Economic  justification  is  not  a  one-time  event  but  rather  is  performed  throughout 
an  acquisition’s  life  cycle  in  order  to  permit  informed  investment  decision  making. 

Earned  value  management 

To  ensure  that  actual  progress  is  being  monitored 
against  expected  progress. 

Earned  value  management  is  a  tool  that  integrates  the  technical,  cost,  and 
schedule  parameters  of  a  contract  and  measures  progress  against  them.  During 
the  planning  phase,  an  integrated  program  baseline  is  developed  by  time 
phasing  budget  resources  for  defined  work.  As  work  is  performed  and  measured 
against  the  baseline,  the  corresponding  budget  value  is  “earned.”  Using  this 
earned  value  metric,  cost  and  schedule  variances,  as  well  as  cost  and  time  to 
complete  estimates,  can  be  determined  and  analyzed. 

Requirements  management 

To  ensure  that  requirements  are  traceable, 
verifiable,  and  controlled. 

Requirements  management  is  the  process  for  ensuring  that  the  requirements  are 
traceable,  verifiable,  and  controlled.  Traceability  refers  to  the  ability  to  follow  a 
requirement  from  origin  to  implementation  and  is  critical  to  understanding  the 
interconnections  and  dependencies  among  the  individual  requirements,  as  well 
as  the  impact  when  a  requirement  is  changed.  Requirements  management 
begins  when  the  contractual  requirements  are  documented  and  ends  when 
system  responsibility  is  transferred  to  the  support  organization. 

Risk  management 

To  ensure  that  risks  are  identified  and 
systematically  mitigated. 

Risk  management  is  the  process  for  identifying  potential  acquisition  problems 
and  taking  appropriate  steps  to  avoid  their  becoming  actual  problems.  Risk 
management  occurs  early  and  continuously  in  the  acquisition  life  cycle. 

16GAO,  Information  Technology:  DOD’s  Acquisition  Policies  and  Guidance  Need  to 
Incorporate  Additional  Best  Practices  and  Controls,  GAO-04-722  (Washington,  D.C.:  July 
30,  2004). 

17GAO-04-722. 
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Business  practice 

Description 

System  quality  measurement 

To  ensure  the  maturity  and  stability  of  system 
products. 

System  quality  measurement  is  the  process  for  understanding  the  maturity  and 
stability  of  the  system  products  being  developed,  operated,  and  maintained  so 
that  problems  can  be  identified  and  addressed  early,  therefore  limiting  their 
overall  impact  on  program  cost  and  schedule.  One  indicator  of  system  quality  is 
the  volume  and  significance  of  system  defect  reports  and  change  proposals. 

Source:  GAO. 

Prior  GAO  Reviews  Have 
Identified  IT  Acquisition 
Management  Weaknesses 
in  DOD’s  Business  System 
Investments 


We  have  previously  reported18  that  DOD  has  not  effectively  managed  a 
number  of  business  system  investments.  Among  other  things,  our  reviews 
of  individual  system  investments  have  identified  weaknesses  in  such  areas 
as  architectural  alignment  and  informed  investment  decision  making, 
which  are  also  the  focus  areas  of  the  Fiscal  Year  2005  National  Defense 
Authorization  Act19  business  system  provisions.  Our  reviews  have  also 
identified  weaknesses  in  other  system  acquisition  and  investment 
management  areas — such  as  EVM,  economic  justification,  requirements 
management,  risk  management,  and  test  management. 


Most  recently,  for  example,  we  reported  that  the  Army’s  approach  to 
investing  about  $5  billion  over  the  next  several  years  in  its  General  Fund 
Enterprise  Business  System,  Global  Combat  Support  System-Army 
Field/Tactical,20  and  Logistics  Modernization  Program  did  not  include 
alignment  with  Army  enterprise  architecture  or  use  a  portfolio-based 
business  system  investment  review  process.21  Moreover,  we  reported  that 
the  Army  did  not  have  reliable  analyses,  such  as  economic  analyses,  to 
support  its  management  of  these  programs.  We  concluded  that  until  the 


18See,  for  example,  GAO,  DOD  Business  Transformation:  Lack  of  an  Integrated  Strategy 
Puts  the  Army’s  Asset  Visibility  System  Investments  at  Risk,  GAO-07-860  (Washington, 
D.C.:  July  27,  2007);  GAO,  Information  Technology:  DOD  Needs  to  Ensure  That  Navy 
Marine  Corps  Intranet  Program  Is  Meeting  Goals  and  Satisfying  Customers,  GAO-07-51 
(Washington,  D.C.:  Dec.  8,  2006);  GAO,  Defense  Travel  System:  Reported  Savings 
Questionable  and  Implementation  Challenges  Remain,  GAO-06-980  (Washington,  D.C.: 
Sept.  26,  2006);  GAO,  DOD  Systems  Modernization:  Uncertain  Joint  Use  and  Marginal 
Expected  Value  of  Military  Asset  Deployment  System  Warrant  Reassessment  of  Planned 
Investment,  GAO-06-171  (Washington,  D.C.:  Dec.  15,  2005);  and  GAO,  DOD  Systems 
Modernization:  Planned  Investment  in  the  Navy  Tactical  Command  Support  System 
Needs  to  Be  Reassessed,  GAO-06-215  (Washington,  D.C.:  Dec.  5,  2005). 

19Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No. 
108-375,  §  332  (2004)  (codified  at  10  U.S.C.  §§  186  and  2,222). 

20Field/tactical  refers  to  Ar  my  units  that  are  deployable  to  locations  around  the  world,  such 
as  Iraq  or  Afghanistan. 

Z1GAO-07-860. 


Page  15 


GAO-08-822  DOD  Business  Systems  Modernization 


Army  adopts  a  business  system  investment  management  approach  that 
provides  for  reviewing  groups  of  systems  and  making  enterprise  decisions 
on  how  these  groups  will  collectively  interoperate  to  provide  a  desired 
capability,  it  runs  the  risk  of  investing  significant  resources  in  business 
systems  that  do  not  provide  the  desired  functionality  and  efficiency. 
Accordingly,  we  made  recommendations  aimed  at  improving  the 
department’s  efforts  to  achieve  total  asset  visibility  and  enhancing  its 
efforts  to  improve  its  control  and  accountability  over  business  system 
investments.  The  department  agreed  with  our  recommendations. 

We  also  reported  that  DON  had  not,  among  other  things,  economically 
justified  its  ongoing  and  planned  investment  in  the  Naval  Tactical 
Command  Support  System  (NTCSS)22  and  had  not  invested  in  NTCSS 
within  the  context  of  a  well-defined  DOD  or  DON  enterprise  architecture. 
In  addition,  we  reported  that  DON  had  not  effectively  performed  key 
measurement,  reporting,  budgeting,  and  oversight  activities  and  had  not 
adequately  conducted  requirements  management  and  testing  activities.  We 
concluded  that,  without  this  information,  DON  could  not  determine 
whether  NTCSS,  as  defined,  and  as  being  developed,  is  the  right  solution 
to  meet  its  strategic  business  and  technological  needs.  Accordingly,  we 
recommended  that  the  department  develop  the  analytical  basis  to 
determine  if  continued  investment  in  the  NTCSS  represents  prudent  use  of 
limited  resources  and  to  strengthen  management  of  the  program, 
conditional  upon  a  decision  to  proceed  with  further  investment  in  the 
program.  The  department  largely  agreed  with  these  recommendations. 

In  addition,  we  reported  that  the  Army  had  not  defined  and  developed  its 
Transportation  Coordinators’  Automated  Information  for  Movements 
System  II  (TC-AIMS  II) — a  joint  services  system  with  the  goal  of  helping  to 
manage  the  movement  of  forces  and  equipment  within  the  United  States 
and  abroad — in  the  context  of  a  DOD  enterprise  architecture.23  We  also 
reported  that  the  Army  had  not  economically  justified  the  program  on  the 
basis  of  reliable  estimates  of  life  cycle  costs  and  benefits  and  had  not 
effectively  implemented  risk  management.  As  a  result,  we  concluded  that 
the  Army  did  not  know  if  its  investment  in  TC-AIMS  II,  as  planned,  is 
warranted  or  represents  a  prudent  use  of  limited  DOD  resources. 
Accordingly,  we  recommended  that  DOD,  among  other  things,  develop  the 
analytical  basis  needed  to  determine  if  continued  investment  in  TC-AIMS 


z2GAO-06-215. 

Z3GAO-06-171. 
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II,  as  planned,  represents  prudent  use  of  limited  defense  resources.  In 
response,  the  department  largely  agreed  with  our  recommendations  and 
has  since  reduced  the  program’s  scope  by  canceling  planned  investments. 


Key  DOD  and  Related 
IT  Acquisition 
Management  Controls 
Have  Not  Been  Fully 
Implemented  on 
GCSS-MC 


DOD  IT-related  acquisition  policies  and  guidance,  along  with  other 
relevant  guidance,  provide  an  acquisition  management  control  framework 
within  which  to  manage  business  system  programs  like  GCSS-MC. 
Effective  implementation  of  this  framework  can  minimize  program  risks 
and  better  ensure  that  system  investments  are  defined  in  a  way  to 
optimally  support  mission  operations  and  performance,  as  well  as  deliver 
promised  system  capabilities  and  benefits  on  time  and  within  budget.  Thus 
far,  GCSS-MC  has  not  been  managed  in  accordance  with  key  aspects  of 
this  framework,  which  has  already  contributed  to  more  than  3  years  in 
program  schedule  delays  and  about  $193  million  in  cost  increases.  These 
IT  acquisition  management  control  weaknesses  include 


•  compliance  with  DOD’s  federated  BEA  not  being  sufficiently 
demonstrated; 

•  expected  costs  not  being  reliably  estimated; 

•  earned  value  management  not  being  adequately  implemented; 

•  system  requirements  not  always  being  effectively  managed,  although 
this  has  recently  improved; 

•  key  program  risks  not  being  effectively  managed;  and 

•  key  system  quality  measures  not  being  used. 


The  reasons  that  these  key  practices  have  not  been  sufficiently  executed 
include  limitations  in  the  applicable  DOD  guidance  and  tools,  and  not 
collecting  relevant  data,  each  of  which  is  described  in  the  applicable 
sections  of  this  report.  By  not  effectively  implementing  these  key  IT 
acquisition  management  controls,  the  program  has  already  experienced 
sizeable  schedule  and  cost  increases,  and  it  is  at  increased  risk  of  (1)  not 
being  defined  in  a  way  that  best  meets  corporate  mission  needs  and 
enhances  performance  and  (2)  costing  more  and  taking  longer  than 
necessary  to  complete. 
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GCSS-MC  Compliance 
with  DOD’s  Federated  BEA 
Has  Not  Been  Sufficiently 
Demonstrated 


DOD  and  federal  guidance24  recognize  the  importance  of  investing  in 
business  systems  within  the  context  of  an  enterprise  architecture.26 
Moreover,  the  2005  National  Defense  Authorization  Act  requires  that 
defense  business  systems  be  compliant  with  DOD’s  federated  BEA.26  Our 
research  and  experience  in  reviewing  federal  agencies  show  that  not 
making  investments  within  the  context  of  a  well-defined  enterprise 
architecture  often  results  in  systems  that  are  duplicative,  are  not  well 
integrated,  are  unnecessarily  costly  to  interface  and  maintain,  and  do  not 
optimally  support  mission  outcomes.27 


24Department  of  Defense  Architecture  Framework,  Version  1.0,  Volume  1  (February  2004); 
GAO,  Information  Technology:  A  Framework  for  Assessing  and  Improving  Enterprise 
Architecture  Management  (Version  1.1),  GAO-03-584G  (Washington,  D.C.:  April  2003); 
Chief  Information  Officer  Council,  A  Practical  Guide  to  Federal  Enterprise  Architecture, 
Version  1.0  (February  2001);  and  Institute  of  Electrical  and  Electronics  Engineers  (IEEE), 
Standard  for  Recommended  Practice  for  Architectural  Description  of  Software-Intensive 
Systems  1471-2000  (Sept.  21,  2000). 

25A  well-defined  enterprise  architecture  provides  a  clear  and  comprehensive  picture  of  an 
entity,  whether  it  is  an  organization  (e.g.,  a  federal  department)  or  a  functional  or  mission 
area  that  cuts  across  more  than  one  organization  (e.g.,  personnel  management).  This 
picture  consists  of  snapshots  of  both  the  enterprise’s  current  or  “As  Is”  environment  and  its 
target  or  “To  Be”  environment,  as  well  as  a  capital  investment  road  map  for  transitioning 
from  the  current  to  the  target  environment.  These  snapshots  consist  of  integrated  “views,” 
which  are  one  or  more  architecture  products  that  describe,  for  example,  the  enterprise’s 
business  processes  and  rules;  infonnation  needs  and  flows  among  functions,  supporting 
systems,  services,  and  applications;  and  data  and  technical  standards  and  structures. 

26DOD  has  adopted  a  federated  approach  for  developing  its  business  mission  area 
enterprise  architecture,  which  includes  the  corporate  BEA  representing  the  thin  layer  of 
DOD-wide  corporate  architectural  policies,  capabilities,  rules,  and  standards;  component 
architectures  (e.g.,  DON  enterprise  architecture);  and  program  architectures  (e.g.,  GCSS- 
MC  architecture). 

2 'See,  for  example,  GAO,  Information  Technology:  FBI  Is  Taking  Steps  to  Develop  an 
Enterprise  Architecture,  but  Much  Remains  to  Be  Accomplished,  GAO-05-363 
(Washington,  D.C.:  Sept.  9,  2005);  GAO,  Homeland  Security:  Efforts  Under  Way  to  Develop 
Enterprise  Architecture,  but  Much  Work  Remains,  GAO-04-777  (Washington,  D.C.:  Aug.  6, 
2004);  GAO,  Information  Technology:  Architecture  Needed  to  Guide  NASA’s  Financial 
Management  Modernization,  GAO-04-43  (Washington,  D.C.:  Nov.  21,  2003);  GAO,  DOD 
Business  Systems  Modernization:  Important  Progress  Made  to  Develop  Business 
Enterprise  Architecture,  but  Much  Work  Remains,  GAO-03-1018  (Washington,  D.C.:  Sept. 
19,  2003);  GAO,  Information  Technology:  DBA  Should  Strengthen  Business  Systems 
Modernization  Architecture  and  Investment  Activities,  GAO-01-631  (Washington,  D.C.: 
June  29,  2001);  and  GAO,  Information  Technology:  INS  Needs  to  Better  Manage  the 
Development  of  Its  En  terprise  Architecture,  GAO/AIMD-OO-212  (Washington,  D.C.: 

Aug.  1,  2000). 
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To  its  credit,  the  program  office  has  followed  DOD’s  BEA  compliance 
guidance.28  However,  this  guidance  does  not  adequately  provide  for 
addressing  all  relevant  aspects  of  BEA  compliance.  Moreover,  DON’s 
enterprise  architecture,  which  is  a  major  component  of  DOD’s  federated 
BEA,  as  well  as  key  aspects  of  DOD’s  corporate  BEA,  have  yet  to  be 
sufficiently  defined  to  permit  thorough  compliance  determinations.  In 
addition,  current  policies  and  guidance  do  not  require  DON  investments  to 
comply  with  its  enterprise  architecture.  This  means  that  the  department 
does  not  have  a  sufficient  basis  for  knowing  if  GCSS-MC  has  been  defined 
to  optimize  DON  and  DOD  business  operations.  Each  of  these  architecture 
alignment  limitations  is  discussed  as  follows: 

•  The  program’s  compliance  assessments  did  not  include  all  relevant 
architecture  products.  In  particular,  the  program  did  not  assess 
compliance  with  the  BEA’s  technical  standards  profile,  which  outlines,  for 
example,  the  standards  governing  how  systems  physically  communicate 
with  other  systems  and  how  they  secure  data  from  unauthorized  access. 
This  is  particularly  important  because  systems,  like  GCSS-MC,  need  to 
employ  common  standards  in  order  to  effectively  and  efficiently  share 
information  with  other  systems.  A  case  in  point  is  GCSS-MC  and  the  Navy 
Enterprise  Resource  Planning  program.29  Specifically,  GCSS-MC  has 
identified  13  technical  standards  that  are  not  in  the  BEA  technical 
standards  profile,  and  Navy  Enterprise  Resource  Planning  has  identified 
25  technical  standards  that  are  not  in  the  profile.  Of  these,  some  relate  to 
networking  protocols,  which  could  limit  information  sharing  between 
these  and  other  systems. 

In  addition,  the  program  office  did  not  assess  compliance  with  the  BEA 
products  that  describe  system  characteristics.  This  is  important  because 
doing  so  would  create  a  body  of  information  about  programs  that  could  be 
used  to  identify  common  system  components  and  services  that  could 
potentially  be  shared  by  the  programs,  thus  avoiding  wasteful  duplication. 
For  example,  our  analysis  of  GCSS-MC  program  documentation  shows 
that  they  contain  such  system  functions  as  receiving  goods,  taking 


"8DOD,  Business  Enterprise  Architecture  Compliance  Guidance  (April  10,  2006). 

29Navy  Enterprise  Resource  Planning  was  initiated  in  2003  to  modernize  and  standardize 
certain  Navy  business  operations,  such  as  financial,  acquisition,  workforce  management, 
supply,  and  maintenance.  Moreover,  according  to  program  officials,  both  Navy  Enterprise 
Resource  Planning  and  GCSS-MC  are  under  the  leadership  of  Navy’s  Program  Executive 
Office  for  Enterprise  Information  Systems,  which  is  responsible  for  developing,  acquiring 
and  deploying  seamless  enterprise-wide  IT  systems. 
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physical  inventories,  and  returning  goods,  which  are  also  system  functions 
cited  by  the  Navy  Enterprise  Resource  Planning  program.  However, 
because  compliance  with  the  BEA  system  products  was  not  assessed,  the 
extent  to  which  these  functions  are  potentially  duplicative  was  not 
considered. 

Furthermore,  the  program  office  did  not  assess  compliance  with  BEA 
system  products  that  describe  data  exchanges  among  systems.  As  we 
previously  reported,  establishing  and  using  standard  system  interfaces  is  a 
critical  enabler  to  sharing  data.30  For  example,  GCSS-MC  program 
documentation  indicates  that  it  is  to  exchange  order  and  status  data  with 
other  systems.  However,  the  program  office  has  not  fully  developed  its 
architecture  product  describing  these  exchanges  and  thus  does  not  have 
the  basis  for  understanding  how  its  approach  to  exchanging  information 
differs  from  that  of  other  systems  that  it  is  to  interface  with.  Compliance 
against  each  of  these  BEA  products  was  not  assessed  because  DOD’s 
compliance  guidance  does  not  provide  for  doing  so  and,  according  to  BTA 
and  program  officials,  some  BEA  and  program-level  architecture  products 
are  not  sufficiently  defined.  According  to  these  officials,  BTA  plans  to 
continue  to  define  these  products  as  the  BEA  evolves. 

•  The  compliance  assessment  was  not  used  to  identify  potential  areas  of 
duplication  across  programs,  which  DOD  has  stated  is  an  explicit  goal  of 
its  federated  BEA  and  associated  investment  review  and  decision-making 
processes.  More  specifically,  even  though  the  compliance  guidance 
provides  for  assessing  programs’  compliance  with  the  BEA  product  that 
defines  DOD  operational  activities,  and  GCSS-MC  was  assessed  for 
compliance  with  this  product,  the  results  were  not  used  to  identify 
programs  that  support  the  same  operational  activities  and  related  business 
processes.  Given  that  the  federated  BEA  is  intended  to  identify  and  avoid 
not  only  duplications  within  DOD  components,  but  also  between  DOD 
components,  it  is  important  that  such  commonality  be  addressed.  For 
example,  program-level  architecture  products  for  GCSS-MC  and  Navy 
Enterprise  Resource  Planning,  as  well  as  two  Air  Force  programs  (Defense 
Enterprise  Accounting  and  Management  System-Air  Force  and  the  Air 
Force  Expeditionary  Combat  Support  System)  show  that  each  supports  at 
least  six  of  the  same  BEA  operational  activities  (e.g.,  conducting  physical 
inventory,  delivering  property,  and  services)  and  three  of  these  four 


30GAO,  DOD  Business  Systems  Modernization:  Progress  in  Establishing  Corporate 
Management  Controls  Needs  to  Be  Replicated  Within  Military  Departments,  GAO-08-705. 
(Washington,  D.C.:  May  15,  2008). 
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programs  support  at  least  18  additional  operational  activities  (e.g., 
performing  budgeting,  managing  receipt,  and  acceptance).  As  a  result, 
these  programs  may  be  investing  in  duplicative  functionality.  Reasons  for 
not  doing  so  were  that  compliance  guidance  does  not  provide  for  such 
analyses  to  be  conducted,  and  programs  have  not  been  granted  access 
rights  to  use  this  functionality. 

•  The  program’s  compliance  assessment  did  not  address  compliance  against 
the  DON’s  enterprise  architecture,  which  is  one  of  the  biggest  members  of 
the  federated  BEA.  This  is  particularly  important  given  that  DOD’s 
approach  to  fully  satisfying  the  architecture  requirements  of  the  2005 
National  Defense  Authorization  Act  is  to  develop  and  use  a  federated 
architecture  in  which  component  architectures  are  to  provide  the 
additional  details  needed  to  supplement  the  thin  layer  of  corporate 
policies,  rules,  and  standards  included  in  the  corporate  BEA.31  As  we 
recently  reported,  the  DON’s  enterprise  architecture  is  not  mature 
because,  among  other  things,  it  is  missing  a  sufficient  description  of  its 
current  and  future  environments  in  terms  of  business  and 
information/data.  However,  certain  aspects  of  an  architecture  nevertheless 
exist  and,  according  to  DON,  these  aspects  will  be  leveraged  in  its  efforts 
to  develop  a  complete  enterprise  architecture.  For  example,  the 
FORCEnet  architecture  documents  DON’s  technical  infrastructure. 
Therefore,  opportunities  exist  for  DON  to  assess  its  programs  in  relation 
to  these  architecture  products  and  to  understand  where  its  programs  are 
exposed  to  risks  because  products  do  not  exist,  are  not  mature,  or  are  at 
odds  with  other  DON  programs.  According  to  DOD  officials,  compliance 
with  the  DON  architecture  was  not  assessed  because  DOD  compliance 
policy  is  limited  to  compliance  with  the  corporate  BEA,  and  the  DON 
enterprise  architecture  has  yet  to  be  sufficiently  developed. 

•  The  program’s  compliance  assessment  was  not  validated  by  DOD  or  DON 
investment  oversight  and  decision-making  authorities.  More  specifically, 
neither  the  DOD  IRBs  nor  the  DBSMC,  nor  the  BTA  in  supporting  both  of 
these  investment  oversight  and  decision-making  authorities,  reviewed  the 
program’s  assessments.  According  to  BTA  officials,  under  DOD’s  tiered 
approach  to  investment  accountability,  these  entities  are  not  responsible 
for  validating  programs’  compliance  assessments.  Rather,  this  is  a 
component  responsibility,  and  thus  they  rely  on  the  military  departments 
and  defense  agencies  to  validate  the  assessments. 


31As  we  recently  reported,  while  the  corporate  BEA  includes  corporate  policies, 
capabilities,  rules,  and  standards,  it  is  still  evolving  and  will  continue  to  add  additional 
details.  See  GAO-08-705. 
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However,  the  DON  Office  of  the  CIO,  which  is  responsible  for 
precertifying  investments  as  compliant  before  they  are  reviewed  by  the 
IRB,  did  not  evaluate  any  of  the  programs’  compliance  assessments. 
According  to  CIO  officials,  they  rely  on  Functional  Area  Managers32  to 
validate  a  program’s  compliance  assessments.  However,  no  DON  policy  or 
guidance  exists  that  describes  how  the  Functional  Area  Managers  should 
conduct  such  validations. 

Validation  of  program  assessments  is  further  complicated  by  the  absence 
of  information  captured  in  the  assessment  tool  about  what  program 
documentation  or  other  source  materials  were  used  by  the  program  office 
in  making  its  compliance  determinations.  Specifically,  the  tool  is  only 
configured,  and  thus  was  only  used,  to  capture  the  results  of  a  program’s 
comparison  of  program  architecture  products  to  BEA  products.  Thus,  it 
was  not  used  to  capture  the  system  products  used  in  making  these 
determinations. 

In  addition,  the  program  office  did  not  develop  certain  program-level 
architecture  products  that  are  needed  to  support  and  validate  the 
program’s  compliance  assessment  and  assertions.  According  to  the 
compliance  guidance,  program-level  architecture  products,  such  as  those 
defining  information  exchanges  and  system  data  requirements  are  not 
required  to  be  used  until  after  the  system  has  been  deployed.  This  is 
important  because  waiting  until  the  system  is  deployed  is  too  late  to  avoid 
the  costly  rework  required  to  address  areas  of  noncompliance.  Moreover, 
it  is  not  consistent  with  other  DOD  guidance,33  which  states  that  program- 
level  architecture  products  that  describe,  for  example,  information 
exchanges,  should  be  developed  before  a  program  begins  system 
development. 

The  limitations  in  existing  BEA  compliance-related  policy  and  guidance, 
the  supporting  compliance  assessment  tool,  and  the  federated  BEA,  puts 
programs  like  GCSS-MC  at  increased  risk  of  being  defined  and 
implemented  in  a  way  that  does  not  sufficiently  ensure  interoperability 
and  avoid  duplication  and  overlap.  We  currently  have  a  review  under  way 
for  the  Senate  Armed  Services  Committee,  Subcommittee  on  Readiness 


32Functional  Area  Managers  are  responsible  for  reducing  the  number  of  DON  applications 
within  specific  portfolios,  such  as  logistics  and  financial  management,  and  reviewing 
program-level  compliance  assertions  before  they  are  submitted  to  the  DON  CIO. 

33Departnrent  of  Defense,  Business  Transformation  Guidance.  Version  1.1.  (July  2007). 
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and  Management  Support,  which  is  examining  multiple  programs’ 
compliance  with  the  federated  BEA. 


Investment  in  GCSS-MC 
Was  Not  Economically 
Justified  Using  Reliable 
Estimates  of  Costs 


The  investment  in  the  first  increment  of  GCSS-MC  has  not  been 
economically  justified  on  the  basis  of  reliable  analyses  of  estimated  system 
costs  over  the  life  of  the  program.  According  to  the  program’s  economic 
analysis,  the  first  increment  will  have  an  estimated  life  cycle  cost  of  about 
$442  million  and  deliver  an  estimated  $1.04  billion  in  risk-adjusted 
estimated  benefits  during  this  same  life  cycle.  This  equates  to  a  net  present 
value  of  about  $688  million.  While  the  most  recent  cost  estimate  was 
derived  using  some  effective  estimating  practices,  it  did  not  make  use  of 
other  practices  that  are  essential  to  having  an  accurate  and  credible 
estimate.  As  a  result,  the  Marine  Corps  does  not  have  a  sufficient  basis  for 
deciding  whether  GCSS-MC,  as  defined,  is  the  most  cost-effective  solution 
to  meeting  its  mission  needs,  and  it  does  not  have  a  reliable  basis  against 
which  to  measure  cost  performance. 


The  Cost  Estimate  Was  Not  A  reliable  cost  estimate  is  critical  to  the  success  of  any  IT  program,  as  it 

Derived  Using  Practices  provides  the  basis  for  informed  investment  decision  making,  realistic 

Necessary  for  an  Accurate  and  budget  formulation  and  program  resourcing,  meaningful  progress 
Credible  Estimate  measurement,  proactive  course  correction,  and  accountability  for  results. 

According  to  the  Office  of  Management  and  Budget  (OMB),34  programs 
must  maintain  current  and  well-documented  cost  estimates,  and  these 
estimates  must  encompass  the  full  life  cycle  of  the  program.  OMB  states 
that  generating  reliable  cost  estimates  is  a  critical  function  necessary  to 
support  OMB’s  capital  programming  process.  Without  reliable  estimates, 
programs  are  at  increased  risk  of  experiencing  cost  overruns,  missed 
deadlines,  and  performance  shortfalls. 


Our  research  has  identified  a  number  of  practices  for  effective  program 
cost  estimating.  We  have  issued  guidance  that  associates  these  practices 


34Office  of  Management  and  Budget,  Circular  No.  A-ll,  Preparation,  Submission,  and 
Execution  of  the  Budget  (Washington,  D.C.:  Executive  Office  of  the  President,  June  2006); 
Circular  No.  A- 130  Revised,  Management  of  Federal  Information  Resources  (Washington, 
D.C.:  Executive  Office  of  the  President,  Nov.  28,  2000);  and  Capital  Programming  Guide: 
Supplement  to  Circular  A-ll,  Part  7,  Preparation,  Submission,  and  Execution  of  the 
Budget  (Washington,  D.C.:  Executive  Office  of  the  President,  June  2006). 
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with  four  characteristics  of  a  reliable  cost  estimate.35  These  four 
characteristic  are  specifically  defined  as  follows: 

•  Comprehensive:  The  cost  estimates  should  include  both  government  and 
contractor  costs  over  the  program’s  full  life  cycle,  from  the  inception  of 
the  program  through  design,  development,  deployment,  and  operation  and 
maintenance,  to  retirement.  They  should  also  provide  a  level  of  detail 
appropriate  to  ensure  that  cost  elements  are  neither  omitted  nor  double 
counted  and  include  documentation  of  all  cost-influencing  ground  rules 
and  assumptions. 

•  Well-documented:  The  cost  estimates  should  have  clearly  defined  purposes 
and  be  supported  by  documented  descriptions  of  key  program  or  system 
characteristics  (e.g.,  relationships  with  other  systems,  performance 
parameters).  Additionally,  they  should  capture  in  writing  such  things  as 
the  source  data  used  and  their  significance,  the  calculations  performed 
and  their  results,  and  the  rationale  for  choosing  a  particular  estimating 
method  or  reference.  Moreover,  this  information  should  be  captured  in 
such  a  way  that  the  data  used  to  derive  the  estimate  can  be  traced  back  to, 
and  verified  against,  their  sources.  The  final  cost  estimate  should  be 
reviewed  and  accepted  by  management  on  the  basis  of  confidence  in  the 
estimating  process  and  the  estimate  produced  by  the  process. 

•  Accurate:  The  cost  estimates  should  provide  for  results  that  are  unbiased 
and  should  not  be  overly  conservative  or  optimistic  (i.e.,  should  represent 
the  most  likely  costs).  In  addition,  the  estimates  should  be  updated 
regularly  to  reflect  material  changes  in  the  program,  and  steps  should  be 
taken  to  minimize  mathematical  mistakes  and  their  significance.  The 
estimates  should  also  be  grounded  in  a  historical  record  of  cost  estimating 
and  actual  experiences  on  comparable  programs. 

•  Credible:  The  cost  estimates  should  discuss  any  limitations  in  the  analysis 
performed  that  are  due  to  uncertainty  or  biases  surrounding  data  or 
assumptions.  Further,  the  estimates’  derivation  should  provide  for  varying 
any  major  assumptions  and  recalculating  outcomes  based  on  sensitivity 
analyses,  and  the  estimates’  associated  risks  and  inherent  uncertainty 
should  be  disclosed.  Also,  the  estimates  should  be  verified  based  on  cross¬ 
checks  using  other  estimating  methods  and  by  comparing  the  results  with 
independent  cost  estimates. 


35GAO,  Cost  Assessment  Guide:  Best  Practices  for  Estim  ating  and  Managing  Program 
Costs,  Exposure  Draft,  GAO-07-1134SP  (Washington,  D.C.:  July  2007). 
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The  $442  million  life  cycle  cost  estimate  for  the  first  increment  reflects 
many  of  the  practices  associated  with  a  reliable  cost  estimate,  including  all 
practices  associated  with  being  comprehensive  and  well-documented,  and 
several  related  to  being  accurate  and  credible.  (See  table  4.)  However, 
several  important  accuracy  and  credibility  practices  were  not  satisfied. 


Table  4:  Summary  of  Cost  Estimating  Characteristics  That  the  Cost  Estimate 
Satisfies 

Characteristic  of  reliable  estimates 

Satisfied?" 

Comprehensive 

Yes 

Well-documented 

Yes 

Accurate 

Partially 

Credible 

Partially 

Source:  GAO  analysis  of  Marine  Corps  data. 

"“Yes”  means  that  the  program  office  provided  documentation  demonstrating  satisfaction  of  the 
criterion.  “Partially”  means  that  the  program  office  provided  documentation  demonstrating  satisfaction 
of  part  of  the  criterion.  “No”  means  that  the  program  office  has  yet  to  provide  documentation 
demonstrating  satisfaction  of  the  criterion. 

The  cost  estimate  is  comprehensive  because  it  includes  both  the 
government  and  contractor  costs  specific  to  development,  acquisition 
(nondevelopment),  implementation,  and  operations  and  support  over  the 
program’s  12-year  life  cycle.  Moreover,  the  estimate  clearly  describes  how 
the  various  subelements  are  summed  to  produce  the  amounts  for  each 
cost  category,  thereby  ensuring  that  all  pertinent  costs  are  included,  and 
no  costs  are  double  counted.  Lastly,  cost-influencing  ground  rules  and 
assumptions,  such  as  the  program’s  schedule,  labor  rates,  and  inflation 
rates  are  documented. 

The  cost  estimate  is  also  well-documented  in  that  the  purpose  of  the  cost 
estimate  was  clearly  defined,  and  a  technical  baseline  has  been 
documented  that  includes,  among  others  things,  the  relationships  with 
other  systems  and  planned  performance  parameters.  Furthermore,  the 
calculations  and  results  used  to  derive  the  estimate  are  documented, 
including  descriptions  of  the  methodologies  used  and  traceability  back  to 
source  data  (e.g.,  vendor  quotes,  salary  tables).  Also,  the  cost  estimate  was 
reviewed  both  by  the  Naval  Center  for  Cost  Analysis  and  the  Office  of  the 
Secretary  of  Defense,  Director  for  Program  Analysis  and  Evaluation, 
which  ensures  a  level  of  confidence  in  the  estimating  process  and  the 
estimate  produced. 
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However,  the  estimate  lacks  accuracy  because  not  all  important  practices 
related  to  this  characteristic  were  satisfied.  Specifically,  while  the  estimate 
is  grounded  in  documented  assumptions  (e.g.,  hardware  refreshment  every 
5  years),  and  periodically  updated  to  reflect  changes  to  the  program,  it  is 
not  grounded  in  historical  experience  with  comparable  programs.  As 
stated  in  our  guide,  estimates  should  be  based  on  historical  records  of  cost 
and  schedule  estimates  from  comparable  programs,  and  such  historical 
data  should  be  maintained  and  used  for  evaluation  purposes  and  future 
estimates  on  other  comparable  programs.  The  importance  of  doing  so  is 
evident  by  the  fact  that  GCSS-MC’s  cost  estimate  has  increased  by  about 
$193  million  since  July  2005,  which  program  officials  attributed  to,  among 
other  things,  schedule  delays,  software  development  complexity,  and  the 
lack  of  historical  data  from  similar  ERP  programs.  While  the  program 
office  did  leverage  historical  cost  data  from  other  ERP  programs, 
including  the  Navy’s  Enterprise  Resource  Planning  Pilot  programs  and 
programs  at  the  Bureau  of  Prisons  and  the  Department  of  Agriculture, 
program  officials  told  us  that  these  programs’  scopes  were  not 
comparable.  For  example,  none  of  the  programs  had  to  utilize  a 
communication  architecture  as  complex  as  the  Marine  Corps,  which 
officials  cited  as  a  significant  factor  in  the  cost  increases  and  a  challenge 
in  estimating  costs. 

The  absence  of  analogous  cost  data  for  large-scale  ERP  programs  is  due  in 
part  to  the  fact  that  DOD  has  not  established  a  standardized  cost  element 
structure  for  ERP  programs  that  can  be  used  to  capture  actual  cost  data. 
According  to  officials  with  the  Defense  Cost  and  Resource  Center,  such 
cost  element  structures  are  needed,  along  with  a  requirement  for  programs 
to  report  on  their  costs,  but  approval  and  resources  have  yet  to  be  gained 
for  either  these  structures  or  the  reporting  of  their  costs.  Until  a 
standardized  data  structure  exists,  programs  like  GCSS-MC  will  continue 
to  lack  a  historical  database  containing  cost  estimates  and  actual  cost 
experiences  of  comparable  ERP  programs.  This  means  that  the  current 
and  future  GCSS-MC  cost  estimates  will  lack  sufficient  accuracy  for 
effective  investment  decision  making  and  performance  measurement. 

Compounding  the  estimate’s  limited  accuracy  are  limitations  in  its 
credibility.  Specifically,  while  the  estimate  satisfies  some  of  the  key 
practices  for  a  credible  cost  estimate  (e.g.,  confirming  key  cost  drivers, 
performing  sensitivity  analyses,36  having  an  independent  cost  estimate 


36Sensitivity  analysis  reveals  how  the  cost  estimate  is  affected  by  a  change  in  a  single 
assumption  or  cost  driver  at  a  time  while  holding  all  other  variables  constant. 
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Benefits  Estimate  Has  Been 
Adjusted  to  Reflect 
Questionable  Assumptions  and 
Limited  Data 


prepared  by  the  Naval  Center  for  Cost  Analysis  that  was  within  4  percent 
of  the  program’s  estimate,  and  conducting  a  risk  analysis  that  showed  a 
range  of  estimated  costs  of  $411  million  to  $523  million),  no  risk  analysis 
was  performed  to  determine  the  program  schedule’s  risks  and  associated 
impact  on  the  cost  estimate.  As  described  earlier  in  this  report,  the 
program  has  experienced  about  3  years  in  schedule  delays  and  recently 
experienced  delays  in  completing  the  solution  design  phase.  Therefore,  the 
importance  of  conducting  a  schedule  risk  analysis  and  using  the  results  to 
assess  the  variability  in  the  cost  estimate  is  critical  for  ensuring  a  credible 
cost  estimate.  Program  officials  agreed  that  the  program’s  schedule  is 
aggressive  and  risky  and  that  this  risk  was  not  assessed  in  determining  the 
cost  estimate’s  variability.  Without  doing  so,  the  program’s  cost  estimate  is 
not  credible,  and  thus  the  program  is  at  risk  of  cost  overruns  as  a  result  of 
schedule  delays. 

Forecasting  expected  benefits  over  the  life  of  a  program  is  also  a  key 
aspect  of  economically  justifying  an  investment.  OMB  guidance37 
advocates  economically  justifying  investments  on  the  basis  of  net  present 
value.  If  net  present  value  is  positive,  then  the  corresponding  benefit-to- 
cost  ratio  will  be  greater  than  1  (and  vice  versa).38  This  guidance  also 
advocates  updating  the  analyses  over  the  life  of  the  program  to  reflect 
material  changes  in  expected  benefits,  costs,  and  risks.  Since  estimates  of 
benefits  can  be  uncertain  because  of  the  imprecision  in  both  the 
underlying  data  and  modeling  assumptions  used,  effects  of  this  uncertainty 
should  be  analyzed  and  reported.  By  doing  this,  informed  investment 
decision  making  can  occur  through  the  life  of  the  program,  and  a  baseline 
can  be  established  against  which  to  compare  the  accrual  of  actual  benefits 
from  deployed  system  capabilities. 

The  original  benefit  estimate  for  the  first  increment  was  based  on 
questionable  assumptions  and  insufficient  data  from  comparable 
programs.  The  most  recent  economic  analysis,  dated  January  2007, 
includes  monetized,  yearly  benefit  estimates  for  fiscal  years  2010-2019  in 


370ffice  of  Management  and  Budget,  Circular  No.  A-94,  Guidelines  and  Discount  Rates 
for  Benefit-Cost  Analysis  of  Federal  Programs  (Oct.  29,  1992). 

38A  benefit-to-cost  ratio  is  calculated  by  dividing  the  present  value  of  benefits  by  the 
present  value  of  costs,  while  net  present  value  is  the  present  value  of  benefits  minus  the 
present  value  of  costs.  Present  value  is  the  worth  of  a  future  stream  of  costs  or  benefits  in 
terms  of  money  paid  or  received  immediately.  Prevailing  interest  rates  provide  the  basis  for 
converting  future  amounts  into  equivalent  present  amounts.  These  interest  rates  serve  as 
discount  rates  in  the  discounting  process — in  the  calculation  of  present  values. 
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three  key  areas — inventory  reductions,  reductions  in  inventory  carrying 
costs,  and  improvements  in  maintenance  processes.  Collectively,  these 
benefits  totaled  about  $2.89  billion  (not  risk-adjusted).  However,  these 
calculations  were  made  using  questionable  assumptions  and  limited  data. 
For  example, 

•  The  total  value  of  the  Marine  Corps  inventory  needed  to  calculate 
inventory  reductions  and  reductions  in  carrying  costs  could  not  be 
determined  because  of  limitations  with  existing  logistic  systems. 

•  The  cost  savings  resulting  from  improvements  in  maintenance  processes 
were  calculated  based  on  assumptions  from  an  ERP  implementation  in  the 
commercial  sector  that,  according  to  program  officials,  is  not  comparable 
in  scope  to  GCSS-MC. 

To  account  for  the  uncertainty  inherent  in  the  benefits  estimate,  the 
program  office  performed  a  Monte  Carlo  simulation.39  According  to  the 
program  office,  this  risk  analysis  generated  a  discounted  and  risk-adjusted 
benefits  estimate  of  $1.04  billion.  As  a  result  of  the  $1.85  billion  adjustment 
to  estimated  benefits,  the  program  office  has  a  more  realistic  benefit 
baseline  against  which  to  compare  the  accrual  of  actual  benefits  from 
deployed  system  capabilities. 


EVM  Has  Not  Been  The  program  office  has  elected  to  implement  EVM,  which  is  a  proven 

Adequately  Implemented  means  for  measuring  program  progress  and  thereby  identifying  potential 

cost  overruns  and  schedule  delays  early,  when  they  can  be  minimized.  In 
doing  so,  it  has  adopted  a  tailored  EVM  approach  that  focuses  on 
schedule.  However,  this  schedule-focused  approach  has  not  been 
effectively  implemented  because  it  is  based  on  a  baseline  schedule  that 
was  not  derived  using  key  schedule  estimating  practices.  According  to 
program  officials,  the  schedule  was  driven  by  an  aggressive  program 
completion  date  established  in  response  to  direction  from  oversight 
entities  to  complete  the  program  as  soon  as  possible.  As  a  result,  they  said 
that  following  these  practices  would  have  delayed  this  completion  date. 
Regardless,  this  means  that  the  schedule  baseline  is  not  reliable,  and 
progress  will  likely  not  track  to  the  schedule. 


39 A  Monte  Carlo  simulation  allows  the  model’s  parameters  to  vary  simultaneously 
according  to  their  associated  probability  distribution.  The  result  is  a  set  of  estimated 
probabilities  of  achieving  alternative  outcomes,  given  the  uncertainty  in  the  underlying 
parameters. 
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A  Tailored  EVM  Approach  Is 
Being  Used  to  Measure 
Program  Progress 


Schedule  Baseline  Was  Not 
Derived  in  Accordance  with 
Key  Estimating  Practices 


The  program  office  has  adopted  a  tailored  approach  to  performing  EVM 
because  of  the  contract  type  being  used.  As  noted  earlier,  the  contract 
types  associated  with  GCSS-MC  integration  and  implementation  vary,  and 
include,  for  example,  firm-fixed-price  contracts40  and  time-and-materials 
contracts.  Under  a  firm-fixed-price  contract,  the  price  is  not  subject  to  any 
adjustment  on  the  basis  of  the  contractor’s  cost  experience  in  performing 
the  contract.  For  a  time-and-materials  contract,  supplies  or  services  are 
acquired  on  the  basis  of  (1)  an  undefined  number  of  direct  labor  hours  that 
are  paid  at  specified  fixed  hourly  rates  and  (2)  actual  cost  for  materials. 

According  to  DOD  guidance,41  EVM  is  generally  not  encouraged  for  firm- 
fixed-price,  level  of  effort,  and  time-and-material  contracts.42  In  these 
situations,  the  guidance  states  that  programs  can  use  a  tailored  EVM 
approach  in  which  an  integrated  master  schedule  (IMS)  is  exclusively  used 
to  provide  visibility  into  program  performance. 

DON  has  chosen  to  implement  this  tailored  EVM  approach  on  GCSS-MC. 

In  doing  so,  it  is  measuring  progress  against  schedule  commitments,  and 
not  cost  commitments,  using  an  IMS  for  each  program  phase.  According  to 
program  officials,  the  IMS  describes  and  guides  the  execution  of  program 
activities.  Regardless  of  the  approach  used,  effective  implementation 
depends  on  having  a  reliable  IMS. 

The  success  of  any  program  depends  in  part  on  having  a  reliable  schedule 
specifying  when  the  program’s  set  of  work  activities  will  occur,  how  long 
they  will  take,  and  how  they  are  related  to  one  another.  As  such,  the 
schedule  not  only  provides  a  road  map  for  the  systematic  execution  of  a 
program,  but  it  also  provides  the  means  by  which  to  gauge  progress, 


40A  firm-fixed-price  contract  provides  for  a  price  that  is  not  subject  to  any  adjustment  on 
the  basis  of  the  contractor’s  cost  experience  in  performing  the  contract.  This  contract  type 
places  upon  the  contractor  maximum  risk  and  full  responsibility  for  all  costs  and  resulting 
profit  or  loss.  It  provides  maximum  incentive  for  the  contractor  to  control  costs  and 
perform  effectively  and  imposes  a  minimum  administrative  burden  upon  the  contracting 
parties. 

41Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide,  (Washington,  D.C.:  October  2006).  See  also  DOD 
Memorandum:  Revision  to  DOD  Earned  Value  Management  Policy  (Washington,  D.C.: 
Mar.  7,  2005). 

42 According  to  DOD  guidance,  EVM  is  generally  not  encouraged  for  firm-fixed-price  and 
time-and-material  contracts,  except  in  certain  situations,  as  determined  by  the  complexity 
of  the  contracted  effort  and  the  inherent  risks  involved. 
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identify  and  address  potential  problems,  and  promote  accountability.  Our 
research  has  identified  nine  practices  associated  with  effective  schedule 
estimating.43  These  practices  are  (1)  capturing  key  activities,  (2) 
sequencing  key  activities,  (3)  assigning  resources  to  key  activities,  (4) 
integrating  key  activities  horizontally  and  vertically,  (5)  establishing  the 
duration  of  key  activities,  (6)  establishing  the  critical  path  for  key 
activities,  (7)  identifying  “float  time”44  between  key  activities,  (8) 
distributing  reserves  to  high-risk  activities,  and  (9)  performing  a  schedule 
risk  analysis. 

The  current  IMS  for  the  solution  design  and  transition-to-build  phase  of 
the  first  increment  was  developed  using  some  of  these  practices.  However, 
it  does  not  reflect  several  practices  that  are  fundamental  to  having  a 
schedule  baseline  that  provides  a  sufficiently  reliable  basis  for  measuring 
progress  and  forecasting  slippages.  To  the  program  office’s  credit,  its  IMS 
captures  and  sequences  key  activities  required  to  complete  the  project, 
integrates  the  tasks  horizontally,  and  identifies  the  program’s  critical  path. 
However,  the  program  office  is  not  monitoring  the  actual  durations  of 
scheduled  activities  so  that  it  can  address  the  impact  of  any  deviations  on 
later  scheduled  activities.  Moreover,  the  schedule  does  not  adequately 
identify  the  resources  needed  to  complete  the  tasks  and  is  not  integrated 
vertically,  meaning  that  multiple  teams  executing  different  aspects  of  the 
program  cannot  effectively  work  to  the  same  master  schedule.  Further,  the 
IMS  does  not  adequately  mitigate  schedule  risk  by  identifying  float  time 
between  key  activities,  introducing  schedule  reserve  for  high-risk 
activities,  or  including  the  results  of  a  schedule  risk  analysis.  See  table  5 
for  the  results  of  our  analyses  relative  to  each  of  the  nine  practices. 


43GAO-07-1 134SP. 

44Float  time  is  the  amount  of  time  an  activity  can  slip  before  affecting  the  critical  path. 


Page  30 


GAO-08-822  DOD  Business  Systems  Modernization 


Table  5:  IMS  Satisfaction  of  Nine  Schedule  Estimating  Key  Practices 


Practice 

Explanation 

Satisfied?8 

GAO  analysis 

Capturing  key 
activities 

The  schedule  should  reflect  all  key  activities  (e.g., 
steps,  events,  outcomes)  as  defined  in  the  program’s 
work  breakdown  structure,  to  include  activities  to  be 
performed  by  both  the  government  and  its  contractors. 

Yes 

The  program’s  schedule  reflects  both 
government  and  contractor  activities,  such  as 
building  and  testing  of  the  software 
components,  as  well  as  key  milestones  for 
measuring  progress. 

Sequencing 
key  activities 

The  schedule  should  be  planned  so  that  it  can  meet 
critical  program  dates.  To  meet  this  objective,  key 
activities  need  to  be  logically  sequenced  in  the  order 
that  they  are  to  be  carried  out.  In  particular,  activities 
that  must  finish  prior  to  the  start  of  other  activities  (i.e., 
predecessor  activities),  as  well  as  activities  that  cannot 
begin  until  other  activities  are  completed  (i.e.,  successor 
activities)  should  be  identified.  By  doing  so, 
interdependencies  among  activities  that  collectively  lead 
to  the  accomplishment  of  events  or  milestones  can  be 
established  and  used  as  a  basis  for  guiding  work  and 
measuring  progress. 

Yes 

The  schedule  includes  the  sequencing  of  key 
activities,  meaning  that  it  includes  both  the 
predecessor  and  successor  activities  and  thus 
establishes  interdependencies  among  the 
activities  that  form  the  basis  for  guiding  work 
and  measuring  progress. 

Establishing 
the  duration 
of  key 
activities 

The  schedule  should  realistically  reflect  how  long  each 
activity  will  take  to  execute.  In  determining  the  duration 
of  each  activity,  the  same  rationale,  historical  data,  and 
assumptions  used  for  cost  estimating  should  be  used 
for  schedule  estimating.  Further,  these  durations  should 
be  as  short  as  possible,  and  they  should  have  specific 
start  and  end  dates.  Excessively  long  periods  needed  to 
execute  an  activity  should  prompt  further  decomposition 
of  the  activity  so  that  shorter  execution  durations  will 
result.  The  schedule  should  be  continually  monitored  to 
determine  when  forecasted  completion  dates  differ  from 
the  planned  dates,  which  can  be  used  to  determine 
whether  schedule  variances  will  affect  downstream 
work. 

Partially 

The  schedule  establishes  the  durations  of  key 
activities  based  on  government  and  contractor 
opinions,  as  well  as  historical  data  from  the 
contractor’s  experience.  However,  the  program 
office  does  not  monitor  the  actual  start  and 
completion  dates  of  work  activities  so  that  the 
impact  of  deviations  on  downstream  scheduled 
work  can  be  proactively  addressed.  Program 
officials  agreed  that  they  have  not  been 
tracking  actual  start  and  end  dates,  but  stated 
that  they  intend  to  do  so  for  future  phases. 

Assigning 
resources  to 
key  activities 

The  schedule  should  reflect  what  resources  (e.g.,  labor, 
material,  and  overhead)  are  needed  to  do  the  work, 
whether  all  required  resources  will  be  available  when 
they  are  needed,  and  whether  any  funding  or  time 
constraints  exist. 

Partially 

The  schedule  allocated  some,  but  not  all, 
resources  to  all  key  activities.  For  example,  it 
did  not  allocate  labor  hours  and  materials. 
Program  officials  stated  that  these  resources 
are  assigned  to  more  detailed  activities  at  the 
team  level.  However,  they  have  yet  to  provide 
adequate  documentation  to  show  that  this  is 
occurring. 
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Practice 

Explanation 

Satisfied?8 

GAO  analysis 

Integrating 
key  activities 
horizontally 
and  vertically 

The  schedule  should  be  horizontally  integrated, 
meaning  that  it  should  link  the  products  and  outcomes 
associated  with  already  sequenced  activities.  These 
links  are  commonly  referred  to  as  “handoffs”  and  serve 
to  verify  that  activities  are  arranged  in  the  right  order  to 
achieve  aggregated  products  or  outcomes.  The 
schedule  should  also  be  vertically  integrated,  meaning 
that  traceability  exists  among  varying  levels  of  activities 
and  supporting  tasks  and  subtasks.  Such  mapping  or 
alignment  among  levels  enables  different  groups  to 
work  to  the  same  master  schedule. 

Partially 

The  schedule  is  horizontally  integrated, 
meaning  that  the  activities  across  the  multiple 
teams  are  arranged  in  the  right  order  to 
achieve  aggregated  products  or  outcomes. 
However,  the  schedule  is  not  vertically 
integrated,  meaning  that  traceability  does  not 
exist  among  varying  levels  of  activities. 

Program  officials  stated  that  team-level 
schedules  can  be  traced  vertically  to  the  IMS, 
but  they  have  not  provided  adequate 
documentation  to  show  that  this  is  occurring. 

Establishing 
the  critical 
path  for  key 
activities 

Using  scheduling  software,  the  critical  path — the  longest 
duration  path  through  the  sequenced  list  of  key 
activities — should  be  identified.  The  establishment  of  a 
program’s  critical  path  is  necessary  for  examining  the 
effects  of  any  activity  slipping  along  this  path.  Potential 
problems  that  might  occur  along  or  near  the  critical  path 
should  also  be  identified  and  reflected  in  the  scheduling 
of  the  time  for  high-risk  activities  (see  next). 

Yes 

The  program’s  critical  path  has  been  defined 
using  scheduling  software  and  includes, 
among  other  things,  the  preparation  for  testing 
activities  and  the  execution  of  six  test 
scenarios. 

Identifying 
float  time 
between  key 
activities 

The  schedule  should  identify  float  time — the  time  that  a 
predecessor  activity  can  slip  before  the  delay  affects 
successor  activities — so  that  schedule  flexibility  can  be 
determined.  As  a  general  rule,  activities  along  the 
critical  path  typically  have  the  least  amount  of  float  time. 

No 

The  program  office  could  not  identify  the 
amount  of  float  time  allocated  to  key  activities 
to  account  for  potential  problems  that  might 
occur  along  or  near  the  critical  path.  Therefore, 
if  the  schedule  for  an  activity  near  the  critical 
path  were  to  slip,  it  is  likely  that  the  delay 
would  impact  the  critical  path. 

Distributing 
reserves  to 
high-risk 
activities 

The  baseline  schedule  should  include  a  buffer  or  a 
reserve  of  extra  time.  Schedule  reserve  for 
contingencies  should  be  calculated  by  performing  a 
schedule  risk  analysis  (see  next).  As  a  general  rule,  the 
reserve  should  be  applied  to  high-risk  activities,  which 
are  typically  found  along  the  critical  path. 

No 

The  program  office  did  not  allocate  schedule 
reserve  for  high-risk  activities  on  the  critical 
path.  Schedule  reserve  should  be  calculated 
by  performing  a  schedule  risk  analysis,  and 
allocating  additional  reserve  for  those  activities 
identified  as  high  risk.  Without  this  reserve, 
any  delays  on  activities  on  the  critical  path 
increase  the  risk  of  delays  to  the  scheduled 
completion  date. 

Performing  a 
schedule  risk 
analysis 

A  schedule  risk  analysis  should  be  performed  using 
statistical  techniques  to  predict  the  level  of  confidence 
in  meeting  a  program’s  completion  date.  This  analysis 
focuses  not  only  on  critical  path  activities  but  also  on 
activities  near  the  critical  path,  since  they  can  potentially 
affect  program  status. 

No 

The  program  office  did  not  perform  a  schedule 
risk  analysis  to  determine  the  level  of 
confidence  in  meeting  the  program’s  activities 
and  its  completion  date. 

Sources:  GAO  analysis  of  Marine  Corps  data. 


“Yes”  means  that  the  program  office  provided  documentation  demonstrating  satisfaction  of  the 
criterion.  “Partially”  means  that  the  program  office  provided  documentation  demonstrating  satisfaction 
of  part  of  the  criterion.  “No”  means  that  the  program  office  has  yet  to  provide  documentation 
demonstrating  satisfaction  of  the  criterion. 


According  to  program  officials,  they  intend  to  begin  monitoring  actual 
activity  start  and  completion  dates  so  that  they  can  proactively  adjust  later 
scheduled  activities  that  are  affected  by  deviations.  However,  they  do  not 
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plan  to  perform  the  three  practices  related  to  understanding  and  managing 
schedule  risk  because  doing  so  would  likely  extend  the  program’s 
completion  date,  and  they  set  this  date  to  be  responsive  to  direction  from 
DOD  and  DON  oversight  entities  to  complete  the  program  as  soon  as 
possible.  In  our  view,  not  performing  these  practices  does  not  allow  the 
inherent  risks  in  meeting  this  imposed  completion  date  to  be  proactively 
understood  and  addressed.  The  consequence  of  omitting  these  practices  is 
a  schedule  that  does  not  provide  a  reliable  basis  for  performing  EVM. 


Requirements  Management  Well-defined  and  managed  requirements  are  recognized  by  DOD  guidance 
Weakness  Was  Recently  38  essential  and  can  be  viewed  as  a  cornerstone  of  effective  system 

Corrected  acquisition.  One  aspect  of  effective  requirements  management  is 

requirements  traceability.45  By  tracing  requirements  both  backward  from 
system  requirements  to  higher  level  business  or  operational  requirements 
and  forward  to  system  design  specifications  and  test  plans,  the  chances  of 
the  deployed  product  satisfying  requirements  are  increased,  and  the  ability 
to  understand  the  impact  of  any  requirement  changes,  and  thus  make 
informed  decision  about  such  changes,  is  enhanced. 

The  program  office  recently  strengthened  its  requirements  traceability.  In 
November  2007,  and  again  in  February  2008,  the  program  office  was 
unable  to  demonstrate  for  us  that  it  could  adequately  trace  its  1,375  system 
requirements  to  both  design  specifications  and  test  documentation. 
Specifically,  the  program  office  was  at  that  time  using  a  tool  called 
DOORS®,  which  if  implemented  properly,  allows  each  requirement  to  be 
linked  from  its  most  conceptual  definition  to  its  most  detailed  definition, 
as  well  as  to  design  specifications  and  test  cases.  In  effect,  the  tool 
maintains  the  linkages  among  requirement  documents,  design  documents, 
and  test  cases  even  if  requirements  change.  However,  the  system 
integration  contractor  was  not  using  the  tool.  Instead  the  contractor  was 
submitting  its  244  work  products,46  accompanied  by  spreadsheets  that 
linked  each  work  product  to  one  or  more  system  requirements  and  test 
cases.  The  program  office  then  had  to  verify  and  validate  the  spreadsheets 


4sDOD,  Defense  Acquisition  Guidebook,  Version  1.0  (Oct.  17,  2004).  Software  Engineering 
Institute,  Software  Acquisition  Capability  Maturity  Model®  version  1.03,  CMU/SEI-2002-TR- 
010  (Pittsburgh,  PA:  March  2002). 

46Work  products  are  the  contractor’s  standardized  business  procedures  and  related  test 
cases,  as  well  as  customized  design  specifications  and  test  cases  for  requirements  that 
cannot  be  met  with  standardized  products,  such  as  for  special  reports  and  external 
interfaces. 
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and  import  and  link  each  work  product  to  the  corresponding  requirement 
and  test  case  in  DOORS.  Because  of  the  sheer  number  of  requirements  and 
work  products  and  its  potential  to  impact  cost,  schedule,  and 
performance,  the  program  designated  this  approach  as  a  medium  risk.  It 
later  closed  the  risk  because  the  proposed  mitigation  strategy  failed  to 
mitigate  it,  and  it  was  realized  as  a  high-priority  program  issue  (i.e., 
problem). 

According  to  program  officials,  this  requirements  traceability  approach 
resulted  in  time-consuming  delays  in  approving  the  design  work  products 
and  importing  and  establishing  links  between  these  products  and  the 
requirements  in  DOORS,  in  part  because  the  work  products  were  not 
accompanied  by  complete  spreadsheets  that  established  the  traceability. 

As  a  result,  about  30  percent  of  the  contractor’s  work  products  had  yet  to 
be  validated,  approved,  and  linked  to  requirements  when  the  design  phase 
was  originally  scheduled  to  be  complete.  Officials  stated  that  the 
contractor  was  not  required  to  use  DOORS  because  it  was  not  experienced 
with  this  tool  and  becoming  proficient  with  it  would  have  required  time 
and  resources,  thereby  increasing  both  the  program’s  cost  and  schedule. 
Ironically,  however,  not  investing  the  time  and  resources  to  address  the 
limitations  in  the  program’s  traceability  approach  contributed  to  recent 
delays  in  completing  the  solution  design  activities,  and  additional 
resources  had  to  be  invested  to  address  its  requirements  traceability 
problems. 

The  program  office  now  reports  that  it  can  trace  requirements  backward 
and  forward.  In  April  2008,  we  verified  this  by  tracing  60  out  of  61 
randomly  sampled  requirements  backward  to  system  requirements  and 
forward  to  approved  design  specifications  and  test  plans.  Program  officials 
explained  that  the  reason  that  we  could  not  trace  the  one  requirement  was 
that  the  related  work  products  had  not  yet  been  approved.  In  addition, 
they  stated  that  there  were  additional  work  products  that  had  yet  to  be 
finalized  and  traced. 

Without  adequate  traceability,  the  risk  of  a  system  not  performing  as 
intended  and  requiring  expensive  rework  is  increased.  To  address  its 
requirements  traceability  weakness,  program  officials  told  us  that  they 
now  intend  to  require  the  contractor  to  use  DOORS  during  the  next  phase 
of  the  program  (build  and  test).  If  implemented  effectively,  the  new 
process  should  address  previous  requirements  traceability  weaknesses 
and  thereby  avoid  a  repeat  of  past  problems. 


Page  34 


GAO-08-822  DOD  Business  Systems  Modernization 


Not  All  Program  Risks 
Have  Been  Adequately 
Managed 


Proactively  managing  program  risks  is  a  key  acquisition  management 
control  and,  if  done  properly,  can  greatly  increase  the  chances  of 
programs  delivering  promised  capabilities  and  benefits  on  time  and  within 
budget.  To  the  program  office’s  credit,  it  has  defined  a  risk  management 
process  that  meets  relevant  guidance.  However,  it  has  not  effectively 
implemented  the  process  for  all  identified  risks.  As  a  result,  these  risks 
have  become  actual  program  problems  that  have  impacted  the  program’s 
cost,  schedule,  and  performance  commitments. 

DOD  acquisition  management  guidance,47  as  well  as  other  relevant 
guidance,48  advocates  identifying  facts  and  circumstances  that  can 
increase  the  probability  of  an  acquisition’s  failing  to  meet  cost,  schedule, 
and  performance  commitments  and  then  taking  steps  to  reduce  the 
probability  of  their  occurrence  and  impact.  In  brief,  effective  risk 
management  consists  of:  (1)  establishing  a  written  plan  for  managing  risks; 
(2)  designating  responsibility  for  risk  management  activities;  (3) 
encouraging  project-wide  participation  in  the  identification  and  mitigation 
of  risks;  (4)  defining  and  implementing  a  process  that  provides  for  the 
identification,  analysis,  and  mitigation  of  risks;  and  (5)  examining  the 
status  of  identified  risks  in  program  milestone  reviews. 

The  program  office  has  developed  a  written  plan  for  managing  risks,  and 
established  a  process  that  together  provide  for  the  above  cited  risk 
management  practices,  and  it  has  followed  many  key  aspects  of  its  plan 
and  process.  For  example, 

The  Program  Manager  has  been  assigned  overall  responsibility  for 
managing  risks.  Also,  individuals  have  been  assigned  ownership  of  each 
risk,  to  include  conducting  risk  analyses,  implementing  mitigation 
strategies,  and  working  with  the  risk  support  team.49 


4‘Department  of  Defense,  Risk  Management  Guide  for  DOD  Acquisition,  6th  Edition, 
Version  1 . 0,  http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-version.pdf 
(accessed  March  13,  2008). 

48Software  Engineering  Institute,  CMMI  for  Acquisition,  Version  1.2,  CMU/SEI-2007-TR- 
017  (Pittsburgh,  PA:  November  2007). 

49The  risk  team  consists  of  (1)  a  senior  risk  advisor  who  provides  risk-related  consultation 
to  management,  (2)  a  risk  manager  who  provides  general  process  and  quality  assurance 
support  for  risk  management,  and  (3)  a  risk  database  administrator  who  issues  risk  reports 
and  maintains  a  risk  database. 
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•  The  plan  and  process  encourage  project-wide  participation  in  the 
identification  and  mitigation  of  risks  by  allowing  program  staff  to  submit  a 
risk  for  inclusion  in  a  risk  database50  and  take  ownership  of  the  risk  and 
the  strategy  for  mitigating  it.  In  addition,  stakeholders  can  bring  potential 
risks  to  the  Program  Manager’s  attention  through  interviews,  where 
potential  risks  are  considered  and  evaluated. 

•  The  program  office  has  thus  far  identified  and  categorized  individual  risks. 
As  of  December  2007,  the  risk  database  contained  27  active  risks — 2  high, 
15  medium,  and  10  low.51 

•  Program  risks  are  considered  during  program  milestone  reviews. 
Specifically,  our  review  of  documentation  for  the  Design  Readiness 
Review,52  a  key  decision  point  during  the  system  development  and 
demonstration  phase  leading  up  to  a  Milestone  C  decision,  showed  that 
key  risks  were  discussed.  Furthermore,  the  Program  Manager  reviews 
program  risks’  status  through  a  risk  watch  list53  and  bimonthly  risk 
briefings. 

However,  the  program  office  has  not  consistently  followed  other  aspects 
of  its  process.  For  example,  it  did  not  perform  key  practices  for  identifying 
and  managing  schedule  risks,  such  as  conducting  a  schedule  risk 
assessment  and  building  in  reserve  time  to  its  schedule.  In  addition, 
mitigation  steps  for  several  key  risks  were  either  not  performed  in 
accordance  with  the  risk  management  strategy,  or  risks  that  were  closed 
as  having  been  mitigated  were  later  found  to  be  actual  program  issues  (i.e., 
problems). 


50The  database  includes  those  active  risks  that  continue  to  require  attention  and  closed 
risks  that  have  either  been  addressed  or  have  become  actual  problems.  The  database  also 
provides  information  such  as  the  risk  owner,  current  risk  level,  likelihood  of  occurrence, 
and  potential  impact  to  the  program’s  cost,  schedule,  and  performance  commitments. 

Risk  levels  of  high,  medium,  low  are  assigned  using  quantitative  measurements  of  the 
probability  of  the  risk  occurring  and  the  potential  impact  to  the  program’s  cost,  schedule, 
and  performance.  Based  on  that  assessment,  a  risk  level  is  assigned  to  represent  the  risk’s 
significance.  High  risks  represent  the  greatest  significance,  medium  risks  represent 
moderate  significance,  and  low  risks  represent  the  least  significant  risks. 

52This  review  determines  whether  the  system’s  design  is  mature  and  stable. 

53The  Program  Manager’s  watch  list  is  a  report  that  captures  the  status,  mitigation  plans, 
and  trends  for  all  high  risks  and  those  medium  risks  that  need  to  be  elevated  to  senior 
managers. 
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For  25  medium  risks  in  the  closed  risk  database,  as  of  February  2008,  4 
were  closed  because  mitigation  steps  were  not  performed  in  accordance 
with  the  strategy  and  the  risks  ultimately  became  actual  issues.  Examples 
from  these  medium  risks  are  as  follows: 

•  In  one  case,  the  mitigation  strategy  was  for  the  contractor  to  deliver 
certain  design  documents  that  were  traced  to  system  requirements  and  to 
do  so  before  beginning  the  solution  build  phase.  The  design  documents, 
however,  were  not  received  in  accordance  with  the  mitigation  strategy. 
Specifically,  program  officials  told  us  that  the  design  documents  contained 
inaccuracies  or  misinterpretations  of  the  requirements  and  were  not 
completed  on  time  because  of  the  lack  of  resources  to  correct  these 
problems.  As  a  result,  the  program  experienced  delays  in  completing  its 
solution  design  activities. 

•  In  another  case,  the  mitigation  strategy  included  creating  the 
documentation  needed  to  execute  the  contract  for  monitoring  the  build 
phase  activities.  However,  the  mitigation  steps  were  not  performed  due  to, 
among  other  things,  delays  in  approving  the  contractual  approach.  As  a 
result,  the  risk  became  a  high-priority  issue  in  February  2008.  According  to 
a  program  issue  report,  the  lack  of  a  contract  to  monitor  system 
development  progress  may  result  in  unnecessary  rework  and  thus 
additional  program  cost  overruns,  schedule  delays,  and  performance 
shortfalls. 

Four  of  the  same  25  medium  risks  were  retired  because  key  mitigation 
steps  for  each  one  were  implemented,  but  the  strategies  proved 
ineffective,  and  the  risks  became  actual  program  issues.  Included  in  these 
4  risks  were  the  following: 

•  In  one  case,  the  program  office  closed  a  risk  regarding  data  exchange  with 
another  DON  system  because  key  mitigation  steps  to  establish  exchange 
requirements  were  fully  implemented.  However,  in  February  2008,  a  high- 
priority  issue  was  identified  regarding  the  exchange  of  data  with  this 
system.  According  to  program  officials,  the  risk  was  mitigated  to  the 
fullest  extent  possible  and  closed  based  on  the  understanding  that 
continued  evaluation  of  data  exchange  requirements  would  be  needed. 
However,  because  the  risk  was  retired,  this  evaluation  did  not  occur. 

•  In  another  case,  a  requirements  management  risk  was  closed  on  the  basis 
of  having  implemented  mitigation  steps,  which  involved  establishing  a 
requirements  management  process,  including  having  complete 
requirements  traceability  spreadsheets.  However,  although  several  of  the 
mitigation  steps  were  not  fully  implemented,  the  risk  was  closed  on  the 
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basis  of  what  program  officials  described  as  an  understanding  reached 
with  the  contractor  regarding  the  requirements  management  process. 
Several  months  later,  a  high-priority  issue  concerning  requirements 
traceability  was  identified  because  the  program  office  discovered  that  the 
contractor  was  not  adhering  to  the  understanding. 

Unless  risk  mitigation  strategies  are  monitored  to  ensure  that  they  are 
fully  implemented  and  that  they  produce  the  intended  outcomes,  and 
additional  mitigation  steps  are  taken  when  they  are  not,  the  program  office 
will  continue  to  be  challenged  in  preventing  risks  from  developing  into 
actual  cost,  schedule,  and  performance  problems. 


Important  Aspects  of 
System  Quality  and 
Program  Maturity  Are  Not 
Being  Measured 


Effective  management  of  programs  like  GCSS-MC  depends  in  part  on  the 
ability  to  measure  the  quality  of  the  system  being  acquired  and 
implemented.  Two  measures  of  system  quality  are  trends  in  (1)  the 
number  of  unresolved  severe  system  defects  and  (2)  the  number  of 
unaddressed  high-priority  system  change  requests. 


GCSS-MC  documentation  recognizes  the  importance  of  monitoring  such 
trends.  Moreover,  the  program  office  has  established  processes  for  (1) 
collecting  and  tracking  data  on  the  status  of  program  issues,  including 
problems  discovered  during  early  test  events,  and  (2)  capturing  data  on 
the  status  of  requests  for  changes  to  the  system.  However,  its  processes  do 
not  provide  the  full  complement  of  data  that  are  needed  to  generate  a 
reliable  and  meaningful  picture  of  trends  in  these  areas.  In  particular,  data 
on  problems  and  change  request  priority  levels  and  closure  dates  are 
either  not  captured  or  not  consistently  maintained.  Further,  program 
office  oversight  of  contractor-identified  issues  or  defects  is  limited. 
Program  officials  acknowledged  these  data  limitations,  but  they  stated  that 
oversight  of  contractor-identified  issues  is  not  their  responsibility.  Without 
tracking  trends  in  key  indicators,  the  program  office  cannot  adequately 
understand  and  report  to  DOD  decision  makers  whether  GCSS-MC’s 
quality  and  stability  are  moving  in  the  right  direction. 

Data  to  Understand  Trends  in  Program  guidance  and  related  best  practices54  encourage  trend  analysis 

Program  Problems  Are  Limited  and  the  reporting  of  system  defects  and  program  problems  as  measures  or 

indicators  of  system  quality  and  program  maturity.  As  we  have  previously 


54GAO,  Year  2000  Computing  Crisis:  A  Testing  Guide,  GAO/AIMD-IO.1.21  (Washington, 
D.C.:  November  1998);  and  IEEE  Std  12207-2008,  Systems  and  software  engineering- 
Software  life  cycle  processes  (Piscataway,  NJ:  2008). 
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reported, 66  these  indicators  include  trends  in  the  number  of  unresolved 
problems  according  to  their  significance  or  priority. 

To  the  program  office’s  credit,  it  collects  and  tracks  what  it  calls  program 
issues,  which  are  problems  identified  by  program  office  staff  or  the  system 
integrator  that  are  process,  procedure,  or  management  related.  These 
issues  are  contained  in  the  program’s  Issues-Risk  Management 
Information  System  (I-RMIS).  Among  other  things,  each  issue  in  I-RMIS  is 
to  have  an  opened  and  closed  date  and  an  assigned  priority  level  of  high, 
medium,  or  low.  In  addition,  the  integration  contractor  tracks  issues  that 
its  staff  identifies  related  to  such  areas  as  system  test  defects.  These  issues 
are  contained  in  the  contractor’s  Marine  Corps  Issue  Tracking  System 
(MCITS).  Each  issue  in  MCITS  is  to  have  a  date  when  it  was  opened  and  is 
to  be  assigned  a  priority  on  a  scale  of  1-5.  According  to  program  officials, 
the  priority  levels  are  based  on  guidance  from  the  Institute  of  Electrical 
and  Electronics  Engineers  (IEEE).  (See  table  6  for  a  description  of  each 
priority  level.) 


Table  6:  MCITS  Priority  Levels 


Priority 

Description 

1 

•  Prevents  the  accomplishment  of  an  essential  capability. 

•  Jeopardizes  safety,  security,  or  other  requirement  designated  critical. 

2 

•  Adversely  affects  the  accomplishment  of  an  essential  capability,  and  no 
work-around  solution  is  known. 

•  Adversely  affects  technical,  cost,  or  schedule  risks  to  the  project  or  to  life 
cycle  support  of  the  system,  and  no  work-around  solution  is  known. 

3 

•  Adversely  affects  the  accomplishment  of  an  essential  capability,  but  a  work¬ 
around  solution  is  known. 

•  Adversely  affects  technical,  cost,  or  schedule  risks  to  the  project  or  to  life 
cycle  support  of  the  system,  but  a  work-around  solution  is  known. 

4 

•  Results  in  user/operator  inconvenience  or  annoyance  but  does  not  affect  a 
required  operational  or  mission-essential  capability. 

•  Results  in  inconvenience  or  annoyance  for  development  or  maintenance 
personnel  but  does  not  prevent  the  accomplishment  of  the  responsibilities  of 
those  personnel. 

5 

•  Any  other  effect. 

Sources:  GCSS-MC  and  IEEE. 


55See,  for  example,  GAO-06-215. 
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Data  to  Understand  Trends  in 
Changes  to  the  System  Are 
Limited 


However,  neither  I-RMIS  nor  MCITS  contain  all  the  data  needed  to  reliably 
produce  key  measures  or  indicators  of  system  quality  and  program 
maturity.  Examples  of  these  limitations  are  as  follows: 

For  I-RMIS,  the  program  office  has  not  established  a  standard  definition  of 
the  priority  levels  used.  Rather,  according  to  program  officials,  each  issue 
owner  is  allowed  to  assign  a  priority  based  on  the  owner’s  definition  of 
what  high,  medium,  and  low  mean.  By  not  using  standard  priority 
definitions  for  categorizing  issues,  the  program  office  cannot  ensure  that  it 
has  an  accurate  and  useful  understanding  of  the  problems  it  is  facing  at 
any  given  time,  and  it  will  not  know  if  it  is  addressing  the  highest  priority 
issues  first. 

For  MCITS,  the  integration  contractor  does  not  track  closure  dates  for  ah 
issues.  For  example,  as  of  April  2008,  over  30  percent  of  the  closed  issues 
did  not  have  closure  dates.  This  is  important  because  it  limits  the 
contractor’s  ability  to  understand  trends  in  the  number  of  high-priority 
issues  that  are  unresolved.  Program  officials  acknowledged  the  need  to 
have  closure  dates  for  ah  closed  issues  and  stated  that  they  intend  to 
correct  this.  If  it  is  not  corrected,  the  program  office  will  not  be  able  to 
create  a  reliable  measure  of  system  quality  and  program  maturity. 

Compounding  the  above  limitations  in  MCITS  data  is  the  program  office’s 
decision  not  to  use  contractor-generated  reports  that  are  based  on  MCITS 
data.  Specifically,  reports  summarizing  MCITS  issues  are  posted  to  a 
SharePoint56  site  for  the  program  office  to  review.  However,  program 
officials  stated  that  they  do  not  review  these  reports  because  the  MCITS 
issues  are  not  their  responsibility,  but  the  contractor’s.  However,  without 
tracking  and  monitoring  contractor-identified  issues,  which  include  such 
things  as  having  the  right  skill-sets  and  having  the  resources  to  track  and 
monitor  issues  captured  in  separate  databases,  the  program  office  is 
missing  an  opportunity  to  understand  whether  proactive  action  is  needed 
to  address  emerging  quality  shortfalls  in  a  timely  manner. 

Program  guidance  and  related  best  practices57  encourage  trend  reporting 
of  change  requests  as  measures  or  indicators  of  system  stability  and 
quality.  These  indicators  include  trends  in  the  number  and  priority  of 


56 An  integrated  software  suite  used  to  facilitate  collaboration,  document  management,  and 
access  to  information  between  the  government  and  contractor. 

J'IEEE  Std  12207-2008,  Systems  and  software  enaineerinqSoftware  life  cycle  processes 
(Piscataway,  NJ:  2008). 
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approved  changes  to  the  system’s  baseline  functional  and  performance 
capabilities  that  have  yet  to  be  resolved. 

To  its  credit,  the  program  office  collects  and  tracks  changes  to  the  system, 
which  can  range  from  minor  or  administrative  changes  to  more  significant 
changes  that  propose  or  impact  important  system  functionality.  These 
changes  can  be  identified  by  either  the  program  office  or  the  contractor, 
and  they  are  captured  in  a  master  change  request  spreadsheet.  Further,  the 
changes  are  to  be  prioritized  according  to  the  level  described  in  table  7, 
and  the  dates  that  change  requests  are  opened  and  closed  are  to  be 
recorded. 


Table  7:  Change  Request  Priorities 


Priority 

Description 

1 

•  Prevents  the  accomplishment  of  an  essential  capability. 

•  Jeopardizes  safety,  security,  or  other  requirement  designated  critical. 

2 

•  Adversely  affects  the  accomplishment  of  an  essential  capability,  and  no 
work-around  solution  is  known. 

•  Adversely  affects  technical,  cost,  or  schedule  risks  to  the  project  or  to  life 
cycle  support  of  the  system,  and  no  work-around  solution  is  known. 

3 

•  Adversely  affects  the  accomplishment  of  an  essential  capability,  but  a  work¬ 
around  solution  is  known. 

•  Adversely  affects  technical,  cost,  or  schedule  risks  to  the  project  or  to  life 
cycle  support  of  the  system,  but  a  work-around  solution  is  known. 

4 

•  Results  in  user/operator  inconvenience  or  annoyance  but  does  not  affect  a 
required  operational  or  mission-essential  capability. 

•  Results  in  inconvenience  or  annoyance  for  development  or  maintenance 
personnel  but  does  not  prevent  the  accomplishment  of  the  responsibilities  of 
those  personnel. 

5 

•  Any  other  effect. 

Sources:  GCSS-MC  and  IEEE. 


However,  the  change  request  master  spreadsheet  does  not  contain  the 
data  needed  to  reliably  produce  key  measures  or  indicators  of  system 
stability  and  quality.  Examples  of  these  limitations  are  as  follows: 

•  The  program  office  has  not  prioritized  proposed  changes  or  managed 
these  changes  according  to  their  priorities.  For  example,  of  the  572  change 
requests  as  of  April  2008,  171  were  assigned  a  priority  level,  and  401  were 
not.  Of  these  171,  132  were  categorized  as  priority  1.  Since  then,  the 
program  office  has  temporarily  recategorized  the  401  change  requests  to 
priority  3  until  each  one’s  priority  can  be  evaluated.  The  program  office 
has  yet  to  establish  a  time  frame  for  doing  so. 


Page  41 


GAO-08-822  DOD  Business  Systems  Modernization 


•  The  dates  that  change  requests  are  resolved  are  not  captured  in  the  master 
spreadsheet.  Rather,  program  officials  said  that  these  dates  are  in  the 
program’s  IMS  and  are  shown  there  as  target  implementation  dates.  While 
the  IMS  does  include  the  dates  changes  will  be  implemented,  these  dates 
are  not  actual  dates,  and  they  are  not  used  to  establish  trends  in 
unresolved  change  requests. 

Without  the  full  complement  of  data  needed  to  monitor  and  measure 
change  requests,  the  program  office  cannot  know  and  disclose  to  DOD 
decision  makers  whether  the  quality  and  stability  of  the  system  are  moving 
in  the  right  direction. 


Conclusions 


DOD’s  success  in  delivering  large-scale  business  systems,  such  as  GCSS- 
MC,  is  in  large  part  determined  by  the  extent  to  which  it  employs  the  kind 
of  rigorous  and  disciplined  IT  management  controls  that  are  reflected  in 
DOD  policies  and  related  guidance.  While  implementing  these  controls 
does  not  guarantee  a  successful  program,  it  does  minimize  a  program’s 
exposure  to  risk  and  thus  the  likelihood  that  it  will  fall  short  of 
expectations.  In  the  case  of  GCSS-MC,  living  up  to  expectations  is 
important  because  the  program  is  large,  complex,  and  critical  to 
supporting  the  department’s  warfighting  mission. 

The  department  has  not  effectively  implemented  a  number  of  essential  IT 
management  controls  on  GCSS-MC,  which  has  already  contributed  to 
significant  cost  overruns  and  schedule  delays,  and  has  increased  the 
program’s  risk  going  forward  of  not  delivering  a  cost-effective  system 
solution  and  not  meeting  future  cost,  schedule,  capability,  and  benefit 
commitments.  Moreover,  GCSS-MC  could  be  duplicating  the  functionality 
of  related  systems  and  may  be  challenged  in  interoperating  with  these 
systems  because  compliance  with  key  aspects  of  DOD’s  federated  BEA 
has  not  been  demonstrated.  Also,  the  program’s  estimated  return  on 
investment,  and  thus  the  economic  basis  for  pursing  the  proposed  system 
solution,  is  uncertain  because  of  limitations  in  how  the  program’s  cost 
estimate  was  derived,  raising  questions  as  to  whether  the  nature  and  level 
of  future  investment  in  the  program  needs  to  be  adjusted.  In  addition,  the 
program’s  schedule  was  not  derived  using  several  key  schedule  estimating 
practices,  which  impacts  the  integrity  of  the  cost  estimate  and  precludes 
effective  implementation  of  EVM.  Without  effective  EVM,  the  program 
cannot  reliably  gauge  progress  of  the  work  being  performed  so  that 
shortfalls  can  be  known  and  addressed  early,  when  they  require  less  time 
and  fewer  resources  to  overcome.  Another  related  indicator  of  progress, 
trends  in  system  problems  and  change  requests,  also  cannot  be  gauged 
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because  the  data  needed  to  do  so  are  not  being  collected.  Collectively, 
these  weaknesses  have  already  helped  to  push  back  the  completion  of  the 
program’s  first  increment  by  more  than  3  years  and  added  about  $193 
million  in  costs,  and  they  are  introducing  a  number  of  risks  that,  if  not 
effectively  managed,  could  further  impact  the  program.  However,  whether 
these  risks  will  be  effectively  managed  is  uncertain  because  the  program 
has  not  always  followed  its  defined  risk  management  process  and,  as  a 
result,  has  allowed  yesterday’s  potential  problems  to  become  today’s 
actual  cost,  schedule,  and  performance  problems. 

While  the  program  office  is  primarily  responsible  for  ensuring  that 
effective  IT  management  controls  are  implemented  on  GCSS-MC,  other 
oversight  and  stakeholder  organizations  share  some  responsibility.  In 
particular,  even  though  the  program  office  has  not  demonstrated  its 
alignment  with  the  federated  BEA,  it  nevertheless  followed  established 
DOD  architecture  compliance  guidance  and  used  the  related  compliance 
assessment  tool  in  assessing  and  asserting  its  compliance.  The  root  cause 
for  not  demonstrating  compliance  thus  is  not  traceable  to  the  program 
office,  but  rather  is  due  to,  among  other  things,  the  compliance  guidance 
and  tool  being  limited,  and  the  program’s  oversight  entities  not  validating 
the  compliance  assessment  and  assertion.  Also,  even  though  the  program’s 
cost  estimate  was  not  informed  by  the  cost  experiences  of  other  ERP 
programs  of  the  same  scope,  the  program  office  is  not  to  blame  because 
the  root  cause  for  this  is  that  the  Defense  Cost  and  Resource  Center  has 
not  maintained  a  standardized  cost  element  structure  for  its  ERP  programs 
and  a  historical  database  of  ERP  program  costs  for  program’s  like  GCSS- 
MC  to  use.  In  contrast,  other  weaknesses  are  within  the  program  office’s 
control,  as  evidenced  by  its  positive  actions  to  address  the  requirements 
traceability  shortcomings  that  we  brought  to  its  attention  during  of  the 
course  of  our  work  and  its  well-defined  risk  management  process. 

All  told,  this  means  that  addressing  the  GCSS-MC  IT  management  control 
weaknesses  require  the  combined  efforts  of  the  various  DOD 
organizations  that  share  responsibility  for  defining,  justifying,  managing, 
and  overseeing  the  program.  By  doing  so,  the  department  can  better 
assure  itself  that  GCSS-MC  will  optimally  support  its  mission  operations 
and  performance  goals  and  will  deliver  promised  capabilities  and  benefits, 
on  time  and  within  budget. 
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To  ensure  that  each  GCSS-MC  system  increment  is  economically  justified 
on  the  basis  of  a  full  and  reliable  understanding  of  costs,  benefits,  and 
risks,  we  recommend  that  the  Secretary  of  Defense  direct  the  Secretary  of 
the  Navy  to  ensure  that  investment  in  the  next  acquisition  phase  of  the 
program’s  first  increment  is  conditional  upon  fully  disclosing  to  program 
oversight  and  approval  entities  the  steps  under  way  or  planned  to  address 
each  of  the  risks  discussed  in  this  report,  including  the  risk  of  not  being 
architecturally  compliant  and  being  duplicative  of  related  programs,  not 
producing  expected  mission  benefits  commensurate  with  reliably 
estimated  costs,  not  effectively  implementing  EVM,  not  mitigating  known 
program  risks,  and  not  knowing  whether  the  system  is  becoming  more  or 
less  mature  and  stable.  We  further  recommend  that  investment  in  all 
future  GCSS-MC  increments  be  limited  if  the  management  control 
weaknesses  that  are  the  source  of  these  risks,  and  which  are  discussed  in 
this  report,  have  not  been  fully  addressed. 

To  address  each  of  the  IT  management  control  weaknesses  discussed  in 
this  report,  we  are  also  making  a  number  of  additional  recommendations. 
However,  we  are  not  making  recommendations  for  the  architecture 
compliance  weaknesses  discussed  in  this  report  because  we  have  a 
broader  review  of  DON  program  compliance  to  the  BEA  and  DON 
enterprise  architecture  that  will  be  issued  shortly  and  will  contain 
appropriate  recommendations. 

To  improve  the  accuracy  of  the  GCSS-MC  cost  estimate,  as  well  as  other 
cost  estimates  for  the  department’s  ERP  programs,  we  recommend  that 
the  Secretary  of  Defense  direct  the  appropriate  organization  within  DOD 
to  collaborate  with  relevant  organizations  to  standardize  the  cost  element 
structure  for  the  department’s  ERP  programs  and  to  use  this  standard 
structure  to  maintain  cost  data  for  its  ERP  programs,  including  GCSS-MC, 
and  to  use  this  cost  data  in  developing  future  cost  estimates. 

To  improve  the  credibility  of  the  GCSS-MC  cost  estimate,  we  recommend 
that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Navy,  through  the 
appropriate  chain  of  command,  to  ensure  that  the  program’s  current 
economic  analysis  is  adjusted  to  reflect  the  risks  associated  with  it  not 
reflecting  cost  data  for  comparable  ERP  programs,  and  otherwise  not 
having  been  derived  according  to  other  key  cost  estimating  practices,  and 
that  future  updates  to  the  GCSS-MC  economic  analysis  similarly  do  so. 

To  enhance  GCSS-MC’s  use  of  EVM,  we  recommend  that  the  Secretary  of 
Defense  direct  the  Secretary  of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  the  program  office  (1)  monitors  the  actual  start 
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and  completion  dates  of  work  activities  performed  so  that  the  impact  of 
deviations  on  downstream  scheduled  work  can  be  proactively  addressed; 

(2)  allocates  resources,  such  as  labor  hours  and  material,  to  all  key 
activities  on  the  schedule;  (3)  integrates  key  activities  and  supporting 
tasks  and  subtasks;  (4)  identifies  and  allocates  the  amount  of  float  time 
needed  for  key  activities  to  account  for  potential  problems  that  might 
occur  along  or  near  the  schedule’s  critical  path;  (5)  performs  a  schedule 
risk  analysis  to  determine  the  level  of  confidence  in  meeting  the  program’s 
activities  and  completion  date;  (6)  allocates  schedule  reserve  for  high-risk 
activities  on  the  critical  path;  and  (7)  discloses  the  inherent  risks  and 
limitations  associated  with  any  future  use  of  the  program’s  EVM  reports 
until  the  schedule  has  been  risk-adjusted. 

To  improve  GCSS-MC  management  of  program  risks,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Secretary  of  the  Navy,  through  the 
appropriate  chain  of  command,  to  ensure  that  the  program  office  (1)  adds 
each  of  the  risks  discussed  in  this  report  to  its  active  inventory  of  risks,  (2) 
tracks  and  evaluates  the  implementation  of  mitigation  plans  for  all  risks, 

(3)  discloses  to  appropriate  program  oversight  and  approval  authorities 
whether  mitigation  plans  have  been  fully  executed  and  have  produced  the 
intended  outcome(s),  and  (4)  only  closes  a  risk  if  its  mitigation  plan  has 
been  fully  executed  and  produced  the  intended  outcome(s). 

To  strengthen  GCSS-MC  system  quality  measurement,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Secretary  of  the  Navy,  through  the 
appropriate  chain  of  command,  to  ensure  that  the  program  office  (1) 
collects  the  data  needed  to  develop  trends  in  unresolved  system  defects 
and  change  requests  according  to  their  priority  and  severity  and  (2) 
discloses  these  trends  to  appropriate  program  oversight  and  approval 
authorities. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Business  Transformation)  and  reprinted  in  appendix 
II,  the  department  stated  that  it  concurred  with  two  of  our 
recommendations  and  partially  concurred  with  the  remaining  five.  In 
general,  the  department  partially  concurred  because  it  said  that  efforts 
were  either  under  way  or  planned  that  will  address  some  of  the 
weaknesses  that  these  recommendations  are  aimed  at  correcting.  For 
example,  the  department  stated  that  GCSS-MC  will  begin  to  use  a  recently 
developed  risk  assessment  tool  that  is  expected  to  assist  programs  in 
identifying  and  mitigating  internal  and  external  risks.  Further,  it  said  that 
these  risks  will  be  reported  to  appropriate  department  decision  makers. 
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We  support  the  efforts  that  DOD  described  in  its  comments  because  they 
are  generally  consistent  with  the  intent  of  our  recommendations  and 
believe  that  if  they  are  fully  and  properly  implemented,  they  will  go  a  long 
way  in  addressing  the  management  control  weaknesses  that  our 
recommendations  are  aimed  at  correcting.  In  addition,  we  have  made  a 
slight  modification  to  one  of  these  five  recommendations  to  provide  the 
department  with  greater  flexibility  in  determining  which  organizations 
should  provide  for  the  recommendation’s  implementation. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees;  the  Director,  Office  of  Management  and  Budget;  the 
Congressional  Budget  Office;  the  Secretary  of  Defense;  and  the 
Department  of  Defense  Office  of  the  Inspector  General.  We  also  will  make 
copies  available  to  others  upon  request.  In  addition,  the  report  will  be 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me 
at  (202)  512-3439  or  hiter@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  GAO  staff  who  made  major  contributions  to  this  report  are 
listed  in  appendix  III. 


Randolph  C.  Hite 
Director,  Information  Technology 
Architecture  and  Systems  Issues 
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Our  objective  was  to  determine  whether  the  Department  of  the  Navy  is 
effectively  implementing  information  technology  management  controls  on 
the  Global  Combat  Support  System-Marine  Corps  (GCSS-MC).  To 
accomplish  this,  we  focused  on  the  first  increment  of  GCSS-MC  relative  to 
the  following  management  areas:  architectural  alignment,  economic 
justification,  earned  value  management,  requirements  management,  risk 
management,  and  system  quality  measurement.  In  doing  so,  we  analyzed  a 
range  of  program  documentation,  such  as  the  acquisition  strategy, 
program  management  plan,  and  Acquisition  Program  Baseline,  and 
interviewed  cognizant  program  officials. 

To  determine  whether  GCSS-MC  was  aligned  with  the  Department  of 
Defense’s  (DOD)  federated  business  enterprise  architecture  (BEA),  we 
reviewed  the  program’s  BEA  compliance  assessments  and  system 
architecture  products,  as  well  as  versions  4.0,  4.1,  and  5.0  of  the  BEA  and 
compared  them  with  the  BEA  compliance  requirements  described  in  the 
Fiscal  Year  2005  National  Defense  Authorization  Act1  and  DOD’s  BEA 
compliance  guidance  and  evaluated  the  extent  to  which  the  compliance 
assessments  addressed  all  relevant  BEA  products.  We  also  determined  the 
extent  to  which  the  program-level  architecture  documentation  supported 
the  BEA  compliance  assessments.  We  obtained  documentation,  such  as 
the  BEA  compliance  assessments  from  the  GCSS-MC  and  Navy  Enterprise 
Resource  Planning  programs,  as  well  as  the  Air  Force’s  Defense  Enterprise 
Accounting  and  Management  System  and  Air  Force  Expeditionary  Combat 
Support  System  programs.  We  then  compared  these  assessments  to 
identify  potential  redundancies  or  opportunities  for  reuse  and  determined 
if  the  compliance  assessments  examined  duplication  across  programs  and 
if  the  tool  that  supports  these  assessments  is  being  used  to  identify  such 
duplication.  In  doing  so,  we  interviewed  program  officials  and  officials 
from  the  Department  of  the  Navy,  Office  of  the  Chief  Information  Officer, 
and  reviewed  recent  GAO  reports  to  determine  the  extent  to  which  the 
programs  were  assessed  for  compliance  against  the  Department  of  the 
Navy  enterprise  architecture.  We  also  interviewed  program  officials  and 
officials  from  the  Business  Transformation  Agency  and  the  Department  of 
the  Navy,  including  the  logistics  Functional  Area  Manager,  and  obtained 
guidance  documentation  from  these  officials  to  determine  the  extent  to 
which  the  compliance  assessments  were  subject  to  oversight  or  validation. 


Donald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No. 
108-375,  §  332  (2004)  (codified  at  10  U.S.C.  §§  186  and  2,222). 
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To  determine  whether  the  program  had  economically  justified  its 
investment  in  GCSS-MC,  we  reviewed  the  latest  economic  analysis  to 
determine  the  basis  for  the  cost  and  benefit  estimates.  This  included 
evaluating  the  analysis  against  Office  of  Management  and  Budget  guidance 
and  GAO’s  Cost  Assessment  Guide?  In  doing  so,  we  interviewed  cognizant 
program  officials,  including  the  Program  Manager  and  cost  analysis  team, 
regarding  their  respective  roles,  responsibilities,  and  actual  efforts  in 
developing  and/or  reviewing  the  economic  analysis.  We  also  interviewed 
officials  at  the  Office  of  Program  Analysis  and  Evaluation  and  Naval 
Center  for  Cost  Analysis  as  to  their  respective  roles,  responsibilities,  and 
actual  efforts  in  developing  and/or  reviewing  the  economic  analysis. 

To  determine  the  extent  to  which  the  program  had  effectively 
implemented  earned  value  management,  we  reviewed  relevant 
documentation,  such  the  contractor’s  monthly  status  reports,  Acquisition 
Program  Baselines,  and  schedule  estimates  and  compared  them  with  DOD 
policies  and  guidance.3  We  also  reviewed  the  program’s  schedule  estimates 
and  compared  them  with  relevant  best  practices4  to  determine  the  extent 
to  which  they  reflect  key  estimating  practices  that  are  fundamental  to 
having  a  reliable  schedule.  In  doing  so,  we  interviewed  cognizant  program 
officials  to  discuss  their  use  of  best  practices  in  creating  the  program’s 
current  schedule. 

To  determine  the  extent  to  which  the  program  implemented  requirements 
management,  we  reviewed  relevant  program  documentation,  such  as  the 
baseline  list  of  requirements  and  system  specifications  and  evaluated  them 
against  relevant  best  practices5  to  determine  the  extent  to  which  the 
program  has  effectively  managed  the  system’s  requirements  and 
maintained  traceability  backward  to  high-level  business  operation 


2Office  of  Management  and  Budget,  Circular  No.  A-94 ,  Guidelines  and  Discount  Rates  for 
Benefit-Cost  Analysis  of  Federal  Program  (Oct.  29,  1992);  GAO,  Cost  Assessment  Guide: 
Best  Practices  for  Estimating  and  Managing  Program  Costs,  Exposure  Draft, 
GAO-07-1134SP  (Washington,  D.C.:  July  2007). 

3Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide,  (Washington,  D.C.:  October  2006).  See  also  DOD 
Memorandum:  Revision  to  DOD  Earned  Value  Management  Policy,  (Washington,  D.C.: 
Mar.  7,  2005). 

4GAO-07-1 134SP. 

5Software  Engineering  Institute,  Software  Acquisition  Capability  Maturity  Model® 
version  1.03,  CMU/SEI-2002-TR-010  (Pittsburgh,  PA:  March  2002). 
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requirements  and  system  requirements,  and  forward  to  system  design 
specifications,  and  test  plans.  To  determine  the  extent  to  which  the 
requirements  were  traceable,  we  randomly  selected  61  program 
requirements  and  traced  them  both  backward  and  forward.  This  sample 
was  designed  with  a  5  percent  tolerable  error  rate  at  the  95  percent  level  of 
confidence,  so  that,  if  we  found  0  problems  in  our  sample,  we  could 
conclude  statistically  that  the  error  rate  was  less  than  5  percent.  Based 
upon  the  weight  of  all  other  factors  included  in  our  evaluation,  our 
verification  of  60  out  of  61  requirements  was  sufficient  to  demonstrate 
traceability.  In  addition,  we  interviewed  program  officials  involved  in  the 
requirements  management  process  to  discuss  their  roles  and 
responsibilities  for  managing  requirements. 

To  determine  the  extent  to  which  the  program  implemented  risk 
management,  we  reviewed  relevant  risk  management  documentation,  such 
as  risk  plans  and  risk  database  reports  demonstrating  the  status  of  the 
program’s  major  risks  and  compared  the  program  office’s  activities  with 
DOD  acquisition  management  guidance6  and  related  best  practices.7  We 
also  reviewed  the  program’s  mitigation  process  with  respect  to  key  risks, 
including  25  medium  risks  in  the  retired  risk  database  that  were  actively 
addressed  by  the  program  office,  to  determine  the  extent  to  which  these 
risks  were  effectively  managed.  In  doing  so,  we  interviewed  cognizant 
program  officials  responsible,  such  as  the  Program  Manager,  Risk 
Manager,  and  subject  matter  experts  to  discuss  their  roles  and 
responsibilities  and  obtain  clarification  on  the  program’s  approach  to 
managing  risks  associated  with  acquiring  and  implementing  GCSS-MC. 

To  determine  the  extent  to  which  the  program  is  collecting  the  data  and 
monitoring  trends  in  the  number  of  unresolved  system  defects  and  the 
number  of  unaddressed  change  requests,  we  reviewed  program 
documentation  such  as  the  testing  strategy,  configuration  management 
policy,  test  defect  reports,  change  request  logs,  and  issue  data  logs.  We 
compared  the  program’s  data  collection  and  analysis  practices  relative  to 


^Department  of  Defense,  Risk  Management  Guide  for  DOD  Acquisition,  6th  Edition, 
Version  1 . 0,  http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-version.pdf 
(accessed  March  13,  2008). 

Software  Engineering  Institute,  CMMI  for  Acquisition,  Version  1.2,  CMU/SEI-2007-TR-017 
(Pittsburgh,  PA:  November  2007). 
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these  areas  to  program  guidance  and  best  practices8  to  determine  the 
extent  to  which  the  program  is  measuring  important  aspects  of  system 
quality.  We  also  interviewed  program  officials  such  as  system  developers, 
relevant  program  management  staff,  and  change  control  managers  to 
discuss  their  roles  and  responsibilities  for  system  quality  measurement. 

We  conducted  our  work  at  DOD  offices  and  contractor  facilities  in  the 
Washington,  D.C.,  metropolitan  area,  and  Triangle,  Va.,  from  June  2007  to 
July  2008,  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objective.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objective. 


8GAO,  Year  2000  Computing  Crisis:  A  Testing  Guide,  GAO/AIMD-IO.1.21  (Washington, 
D.C.:  November  1998);  and  IEEE  Std  12207-2008,  Systems  and  software  engineering- 
Software  life  cycle  processes  (Piscataway,  NJ:  2008). 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


ACQUISITION 

TECHNOLOGY 

AND  LOGISTICS  jyy  £1  2008 


Mr.  Randolph  C.  Hite 

Director.  Information  Technology  Architecture  and  Systems  Issues 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Hite: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  GAO- 
08-822,  “DOD  BUSINESS  SYSTEMS  MODERNIZATION:  Key  Marine  Corps  System 
Acquisition  Needs  to  be  Belter  Justified,  Defined  and  Managed,”  dated  June  9,  2008 
(GAO  Code  310648).  Detailed  comments  on  the  report  recommendations  are  enclosed. 

DoD  concurred  with  two  recommendations  and  partially  concurred  with  the  remaining 
five.  Overall,  the  Department  has  already  taken  steps  to  address  some  of  GAO’s  findings, 
including  updating  guidance  for  Business  Enterprise  Architecture  (BEA)  5.0  compliance, 
Further,  the  Department’s  Business  Capability  Lifecycle  (BCL)  is  expected  to  provide 
stakeholders  at  the  program-.  Component-,  and  governance-levels  with  additional  visibility  into 
program  risks  via  the  Enterprise  Risk  Assessment  Methodology  (ERAM)  as  well  as  further 
visibility  into  existing  management  control  issues.  BCL  will  also  support  (he  collection  of  cost 
data  for  Enterprise  Resource  Planning  (ERP)  for  programs’  future  cost  estimation  purposes. 

We  welcome  the  GAO’s  insight  on  the  Department’s  progress  with  its  business 
transformation  efforts  and  continue  to  value  our  partnership. 


Sincerely, 


Deputy  Under  Secretary  of  Defense 
(Business  Transformation) 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  DATED  JUNE  9,  2008 
GAO-08-822  (GAO  CODE  310648)” 


“DOD  BUSINESS  SYSTEMS  MODERNIZATION:  KEY  MARINE  CORPS 
SYSTEM  ACQUISITION  NEEDS  TO  BE  BETTER  JUSTIFIED,  DEFINED, 
AND  MANAGED 


DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy  to  ensure  that  investment  in  the  next  acquisition  phase  of  the 
program’s  first  increment  is  conditional  upon  fully  disclosing  to  program  oversight  and 
approval  entities  the  steps  underway  or  planned  to  address  each  of  the  risks  discussed  in 
this  report,  including  the  risk  of  not  being  architecturally  compliant  and  being  duplicative 
of  related  programs,  not  producing  expected  mission  benefits  commensurate  with  reliably 
estimated  costs,  not  effectively  implementing  earned  value  management  (EVM),  not 
mitigating  known  program  risks,  and  not  knowing  whether  the  system  is  becoming  more 
or  less  mature  and  stable.  (Page  54/GAO  Draft  Report) 

DOD  RESPONSE:  Partially  Concur. 

The  Department  is  already  taking  steps  to  promote  greater  visibility  into  program  risks 
through  its  implementation  of  the  Business  Capability  Lifecycle  (BCL).  Per  a  May  1 8, 
2007  memo  from  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (AT&L)/Vice  Chair  of  the  Defense  Business  Systems  Management  Committee 
(DBSMC),  the  Global  Combat  Support  Systems  -  Marine  Corps  (GCSS-MC)  program  is 
one  of  the  programs  that  will  begin  to  use  the  Enterprise  Risk  Assessment  Methodology 
(ERAM)  for  the  duration  of  its  lifecycle,  which  will  assist  the  program  in  identifying  and 
mitigating  internal  and  external  risks.  ERAM  findings  are  shared  with  appropriate  team 
members  and  decision  makers  within  the  Component  as  well  as  Investment  Review 
Board  (IRB)  and  DBSMC  leadership,  and  programs  utilize  ERAM  findings  to  create  risk 
mitigation  plans  as  needed.  Given  that  future  development  of  the  GCSS-MC  capabilities 
will  be  achieved  via  an  incremental  approach  as  part  of  the  Department’s  BCE  process,  it 
is  envisioned  that  any  emerging  risk  issues  will  be  identified  and  addressed. 

Additionally,  the  GCSS-MC  Program  Office  is  promoting  greater  visibility  of  risks  at  the 
program  level  by  developing  and  using  a  database  to  track  emerging  risks  for  mitigation 
in  future  increments,  and  the  program  manager  is  kept  apprised  of  the  status  of  these  risks 
through  a  watch  list  and  regular  briefings  by  the  risk  owners. 
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The  program  is  also  taking  several  steps  to  ensure  that  the  program  is  architecturally 
compliant  in  accordance  with  current  DoD  guidance  and  policies.  As  the  Business 
Enterprise  Architecture  (BEA)  evolves,  the  GCSS-MC  program  has  continued  to  update 
system  maps  as  required  as  part  of  the  acquisition  review  process.  Additionally,  as  part 
of  the  Marine  Corps’  overall  Logistics  Modernization  effort,  a  Logistics  Operational 
Architecture  (LOG  OA)  was  developed  to  guide  future  development  of  Logistics  IT 
systems  used  by  the  Marine  Air  Ground  Task  Forces  (MAGTFs)  and  the  supporting 
establishment  organizations.  This  architecture  has  been  mapped  to  the  BEA,  the 
USTRANSCOM  Deployment  and  Distribution  Enterprise  Architecture  and  efforts  are 
ongoing  to  develop  a  Joint  Army-Marine  Corps  Integrated  Architecture  in  support  of 
logistics  operations.  GCSS-MC,  as  the  centerpiece  of  the  Logistics  Modernization  effort, 
has  been  aligned  to  the  LOG  OA  to  provide  a  basis  for  identifying  capability  gaps  and 
overlaps  in  architecture  that  are  not  being  used  to  develop  future  capability  requirements. 

The  Business  Transformation  Agency  (BTA)  will  also  continue  to  revise  the  BEA 
compliance  guidance  and  further  define  program-level  architecture  products  to  provide 
the  Components  with  clearer  requirements.  For  example,  the  latest  issuance  of  the  BEA 
compliance  guidance  in  May  2008  (attached)  updated  the  BEA  compliance  requirements 
to  enable  Components  to  assert  to  the  BEA  5.0  data  synonym  and  attributes  that  were 
developed  in  support  of  Enterprise  Data  Initiatives,  and  to  enable  the  IRBs  to  enforce 
program  compliance  to  a  more  defined  set  of  requirements  that  promote  interoperability 
and  data  standardization.  The  new  guidance  document  will  better  assist  the  Components 
as  they  map  their  programs  to  BEA  5.0  requirements  in  future  increments. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy  to  ensure  that  investment  in  all  future  Global  Combat  Support 
Systems-Marine  Corps  (GCSS-MC)  increments  be  limited  if  the  management  control 
weaknesses  that  arc  the  source  of  these  risks,  and  which  are  discussed  in  this  report,  have 
not  been  fully  addressed.  (Page  54/GAO  Draft  Report) 

DOD  RESPONSE:  Partially  concur. 

The  Department  is  already  taking  steps  to  address  some  of  the  management  control  issues 
GAO  noted  in  the  report.  As  previously  discussed,  the  BEA  compliance  guidance  was 
recently  updated  to  provide  Components  with  additional  clarity  on  requirements  for 
asserting  compliance  to  BEA  5.0.  In  addition,  the  implementation  ol'BCL  is  expected  to 
provide  the  program  offices.  Components,  and  IRB/DBSMC  leadership  with  enhanced 
visibility  of  risks.  This  visibility  will  allow  the  Department  to  examine  the  root  causes  of 
identified  risks  and  take  action  as  needed  to  make  improvements  in  its  management 
controls  as  business  transformation  efforts  continue. 

Going  forward,  the  GCSS-MC  Program  Office  will  address  the  correction  of  all  identified 
weaknesses  prior  to  seeking  investment  decisions  for  future  increments  in  accordance 
with  all  existing  Department  of  the  Navy  and  Department  of  Defense  guidance  and 
policies,  including  IRB  policy  and  Guidance. 
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RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Director  of  the  Office  of  Program  Analysis  and  Evaluation  to  ensure  the  Defense  Cost 
and  Resource  Center,  in  collaboration  with  other  relevant  organizations,  standardize  the 
cost  element  structure  for  the  department’s  Enterprise  Resource  Planning  (ERP)  programs 
and  to  use  this  standard  structure  to  maintain  cost  data  for  its  ERP  programs,  including 
GCSS-MC,  and  to  use  this  cost  data  in  developing  future  cost  estimates.  (Page  55/GAO 
Draft  Report) 

POD  RESPONSE:  Partially  Concur 

The  Department  concurs  with  GAO’s  recommendation  that  actual  cost  data  for  ERP 
programs  should  be  maintained  for  use  in  developing  future  cost  estimates.  Assembling 
this  data  would  be  best  achieved  by  taking  steps  to  standardize  the  Work  Breakdown 
Structure  (WBS)  for  ERPs.  The  B'l’A  will  support  the  Office  of  AT&L/Acquisition 
Resources  and  Analysis  (ARA),  the  organization  responsible  for  issuing  DoD’s  handbook 
on  work  breakdown  structures  for  defense  materiel  items  (MIL-HDBK-881A),  in 
developing  the  standard  WBS  for  ERP  systems. 

In  addition,  the  Department  intends  to  track  and  maintain  cost  data  for  ERPs  through 
BCL’s  Integrated  Management  Information  Environment  (IMIE).  Since  BCL  unifies 
reporting  requirements  of  the  Defense  Acquisition  System  (DAS)  5000-scrics  policies, 
the  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170,  and  the  1RB  Concept 
of  Operations  (CONOPS)  for  business  systems,  the  required  information  is  presented  to 
the  IRB/DBSMC  governance  structure,  including  the  program’s  economic  analysis.  As 
ERPs  undergo  the  BCL  process  and  more  business  cases  are  submitted  to  the  IRBs,  cost 
data  will  be  captured  in  IMIE  for  use  in  developing  future  cost  estimates,  and  in 
developing  economic  analysis  guidance  for  both  acquisition  and  Clinger-Cohen  Act 
compliance  decisions. 

RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary'  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that  the 
program’s  current  economic  analysis  is  adjusted  to  reflect  the  risks  associated  with  it  not 
reflecting  cost  data  for  comparable  ERP  programs,  and  otherwise  not  having  been  derived 
according  to  other  key  costing  estimating  practices,  and  that  future  updates  to  the  GCSS- 
MC  economic  analysis  similarly  do  so.  (Page  55/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur 

To  date,  economic  analyses  for  GCSS-MC  have  leveraged  historical  cost  data  from  ERP 
programs;  however  this  is  not  analogous  historical  cost  data  for  large-scale  ERP  programs 
with  similar  architectural  complexities.  Per  DoD’s  response  to  Recommendation  3,  the 
Department  intends  to  track  and  maintain  cost  data  for  ERPs  through  the  BCL’s  IMIE. 
which  will  assist  programs  in  developing  future  economic  analyses,  and  a  standard  WBS 
for  ERP  systems  will  be  developed  by  AT&L/ARA  and  BTA. 
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Through  the  Naval  Center  for  Cost  Analysis  (NCCA),  the  Department  of  Navy  conducted 
a  risk  analysis  of  the  GCSS-MC  estimate,  which  included  a  schedule  component.  This 
information  was  provided  as  part  of  the  data  collected  through  NCCA,  and  cited  in  the 
Department  of  the  Navy  Cost  Analysis  Improvement  Group  (DON  CAIG)  memo. 

Moving  forward,  the  program  will  also  include  risk  analysis,  as  a  point  comparison  to  that 
developed  through  the  DON  CAIG.  The  program  is  currently  preparing  a  cost  estimate  in 
support  of  the  GCSS-MC  Block  1  Milestone  C  and  will  ensure  that  these  and  future  costs 
are  adjusted  to  reflect  risks  associated  with  the  lack  of  comparable  historical  data. 

RECOMMENDATION  5:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that  the 
program  office:  ( 1 )  monitors  the  actual  start  and  completion  dates  of  work  activities 
performed  so  that  the  impact  of  deviations  on  downstream  scheduled  work  can  he 
proactively  addressed;  (2)  allocates  resources,  such  as  labor  hours  and  material,  to  all  key 
activities  on  the  schedule;  (3)  integrates  key  activities  and  supporting  tasks  and  sub-tasks; 
(4)  identifies  and  allocates  the  amount  of  float  time  needed  for  key  activities  to  account 
for  potential  problems  that  might  occur  along  or  near  the  schedule's  critical  path;  (5) 
performs  a  schedule  risk  analysis  to  determine  the  level  of  confidence  in  meeting  the 
program’s  activities  and  completion  date;  (6)  allocates  schedule  reserve  for  high  risk 
activities  on  the  critical  path;  and  (7)  discloses  the  inherent  risks  and  limitations 
associated  with  any  future  use  of  the  program’s  EVM  reports  until  the  schedule  has  been 
risk-adjusted.  (Page  55/GAO  Draft  Report) 

POD  RESPONSE:  Concur. 

The  Department’s  EVM  policy  recognizes  that  the  preparation  of  an  integrated  master 
schedule  (IMS)  is  a  best  practice,  and  requires  the  preparation  of  an  IMS  whenever  EVM 
compliance  is  required.  GCSS-MC  is  currently  rcbaselining  its  own  integrated  master 
schedule  (IMS)  to  adjust  for  schedule  risk.  The  rebaselincd  GCSS-MC  IMS  will  be  used 
to: 

•  Provide  a  baseline  for  monitoring  and  reporting  planned  versus  actual  start  and 
completion  dates  of  work  activities  performed,  so  that  the  impact  of  deviations  on 
downstream  scheduled  work  can  be  proactively  addressed; 

•  Allocate  resources  to  activities; 

•  Integrate  key  activities  and  supporting  tasks  and  sub-tasks; 

•  Identify  and  allocate  the  amount  of  float  time  needed  for  key  activities; 

•  Allocate  schedule  reserve  for  high  risk  activities  on  the  critical  path. 

As  the  schedule  is  rebaselined,  GCSS-MC  will  perform  a  schedule  risk  analysis  to 
determine  the  level  of  confidence  in  meeting  the  program’s  activities  and  completion 
dates.  The  program  will  disclose  the  inherent  risk  and  limitations  associated  with  the 
program’s  EVM  reports  until  the  schedule  has  been  rebaselined  and  risk  adjusted. 
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RECOMMENDATION  6:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that  the 
program  office:  (I)  adds  each  of  the  risks  discussed  in  this  report  to  its  active  inventory  of 
risks;  (2)  tracks  and  evaluates  the  implementation  of  mitigation  plans  for  all  risks;  (3) 
discloses  to  appropriate  program  oversight  and  approval  authorities  whether  mitigation 
plans  have  been  fully  executed  and  have  produced  the  intended  outcome(s);  and  (4)  only 
closes  a  risk  if  its  mitigation  plan  has  been  fully  executed  and  produced  the  intended 
outcome(s).  (Page  55/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur. 

GCSS-MC  has  addressed  the  majority  of  these  risk  factors  through  the  240-plus  risks  in 
the  risk  database,  although  not  necessarily  in  the  general  wording  of  the  GAO  report,  but 
instead  has  addressed  them  under  a  broader  range  of  risk  topics  in  details.  The  Program 
Office  will,  however,  re-examine  the  risks  in  the  database  to  ensure  that  the  risks  in  this 
report  are  covered.  In  addition,  program  risks  are  discussed  during  program  milestone 
reviews  and  regularly  reviewed  by  the  program  manager. 

RECOMMENDATION  7:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that  the 
program  office:  (1)  collects  the  data  needed  to  develop  trends  in  unresolved  system 
defects  and  change  requests  according  to  their  priority  and  severity;  and  (2)  discloses 
these  trends  to  appropriate  program  oversight  and  approval  authorities. 

POD  RESPONSE:  Concur 

The  GCSS-MC  program  office  is  currently  working  with  an  independent  software 
measurement  analysis  company  to  collect  the  data  needed  to  develop  trends  in  unresolved 
system  defects  and  change  requests  according  to  their  priority  and  severity  levels.  Results 
will  be  reported  to  appropriate  program  oversight  authorities  on  a  monthly  basis, 
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